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ABSTRACT 


Command,  Control,  Communications,  Computers,  and  Intelligence  (C4I)  systems, 
each  originally  designed  to  address  a  single  warfighting  function,  have  been  assembled 
into  an  interdependent  C4I  System  of  Systems  (SoS).  The  C4I  SoS  continues  to  evolve, 
without  overarching  capabilities-based  performance  requirements.  Without 
requirements,  there  is  no  practical,  repeatable,  and  objective  process  to  assess  changes  to 
the  SoS.  This  project  applied  a  disciplined  systems  engineering  process  to  design  a  Joint 
C4I  Capability  Certification  Measures  (JC3M)  system.  JC3M  can  be  used  to  define 
performance  measures  for  a  C4I  SoS,  determine  baseline  SoS  performance,  assess 
proposed  SoS  changes,  and  monitor  SoS  performance.  Modeling  and  simulation  tools 
were  used  to  project  the  performance  of  three  existing  alternatives  and  two  new 
alternatives.  A  Life  Cycle  Cost  Estimate  (LCCE)  was  generated  for  each  alternative.  An 
Analysis  of  Alternatives  compared  performance  and  cost.  The  Joint  Test  and  Evaluation 
Methodology  Capability  Test  Methodology  (JTEM  CTM)  was  projected  to  provide 
slightly  better  performance  than  other  alternatives,  at  the  median  LCCE.  The  results 
were  a  recommendation  to  monitor  JTEM  CTM  as  it  completes  development,  and 
employ  the  JTEM  CTM  in  a  C4I  SoS  evaluation  to  confirm  its  estimated  cost  and 
performance. 


v 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


vi 


TABLE  OF  CONTENTS 


I.  INTRODUCTION . 1 

A.  BACKGROUND . 1 

B.  PROBLEM  STATEMENT . 6 

C.  OPERATING  FORCES  NEED  FOR  JC3M . 7 

1.  Example  1  -  Voice  Switch  Fielding . 7 

2.  Example  2  -  Gateway  Exchange  Interoperability . 9 

D.  THE  SYSTEMS  ENGINEERING  DESIGN  PROCESS . 9 

1.  Overview . 9 

2.  JC3M  Systems  Engineering  Process  Phases . 11 

a.  Problem  Refinement  Phase . 11 

b.  Design  Alternatives  Phase . 12 

c.  Modeling  and  Simulation  Phase . 13 

d.  Analysis  of  Alternatives  Phase . 14 

e.  Final  Recommendation  Phase. . 1 5 

f.  Assess  Performance  Phase . 16 

II.  PROBLEM  REFINEMENT . 17 

A.  NEEDS  ANALYSIS . 18 

1.  Stakeholder  Identification . 18 

2.  Stakeholder  Initial  Requirements . 19 

B.  REQUIREMENTS  GENERATION  AND  ANALYSIS . 20 

1.  Requirements  Identification . 20 

2.  Requirements  Analysis . 21 

3.  JC3M  Inputs  and  Outputs . 22 

C.  VALUE  SYSTEM  DESIGN . 23 

1 .  Value  System  Modeling . 23 

2.  Value  Hierarchy . 24 

3.  Critical  Evaluation  Measures . 25 

III.  DESIGN  ALTERNATIVES . 27 

A.  ALTERNATIVE  GENERATION . 27 

1.  Known  Alternatives . 28 

a.  Federation  of  Systems . 28 

b.  Marine  Air  Ground  Task  Force  Command,  Control, 

Communications,  Computers,  and  Intelligence  Capability 
Certification  Test . 32 

c.  Joint  Test  and  Evaluation  Methodology  Capability  Test 

Methodology . 34 

2.  Generating  New  Alternatives . 36 

3.  Morphological  Box  Process . 37 

B.  FEASIBILITY  SCREENING . 40 

1.  Alternative  Refinement . 42 

2.  JC3M  Changes . 44 

a.  Systems  Capabilities  Review  Alternative . 46 

b.  Functional  Capabilities  Board  Alternative . 48 

vii 


c.  Differences  between  SCR  and  FCB . 52 

C.  ALTERNATIVE  SCORING  CRITERIA . 54 

IV.  MODELING  AND  SIMULATION . 57 

A.  JC3M  USES  OF  MODELING  AND  SIMULATION . 57 

B.  JC3M  MODELING  TOOLS . 59 

1.  CORE . 59 

2.  POW-ER . 59 

3.  ARENA . 62 

C.  MODELING  AND  SIMULATION  APPROACH . 67 

1.  Select  a  Baseline  System . 67 

2.  Select  a  Candidate  C4I  Test  Architecture  as  a  Baseline  SoS . 68 

3.  Assemble  a  Data  Set  for  the  FEDOS  System . 69 

4.  Model,  Simulate,  and  Validate  the  Baseline  Processes . 70 

5.  Map  the  Alternatives  to  the  Baseline  Processes . 71 

6.  Create  Data  Sets  for  Each  JC3M  Alternative . 72 

7.  Build  POW-ER,  ARENA,  and  CORE  Model  for  Each 

Alternative . 72 

8.  Run  the  Models  and  Capture  Simulation  Results . 73 

D.  BASELINE  MODEL  DEVELOPMENT . 73 

1.  Baseline  C4I  Architecture . 73 

2.  Baseline  System  of  System  Capabilities . 75 

3.  Defining  “Old”  versus  “New”  Capabilities . 75 

E.  SIMULATION  RESULTS . 77 

1.  Arena  Simulation  Results . 77 

2.  Power  Simulation  Results . 79 

3.  Results  of  Model  Validation . 79 

F.  SUMMARY . 80 

V.  LIFE  CYCLE  COSTS . 81 

A.  JC3M  LIFE  CYCLE  COST  OVERVIEW . 81 

1.  Introduction . 81 

a.  Development  and  Implementation  Phase. . 82 

b.  Operation  and  Support  Phase. . 83 

c.  Transition  and  Retirement  Phase . 83 

2.  Life  Cycle  Cost  Elements . 84 

a.  Development  and  Implementation  Costs . 84 

b.  Operation  and  Support  Costs . 84 

c.  Transition  and  Retirement  Costs . 84 

3.  Cost  Estimation  Assumptions  and  Constraints . 85 

B.  COST  ESTIMATION  METHODS  CONSIDERED . 85 

C.  JC3M  COST  ESTIMATION  APPROACH . 87 

D.  COST  DATA  COLLECTION . 88 

E.  ANALYSIS  OF  COST  DATA . 89 

1.  Development  Costs . 89 

2.  Implementation  Costs . 90 

3.  Results  of  Cost  Data  Analysis . 91 

viii 


VI.  ANALYSIS  OF  ALTERNATIVES . 93 

A.  METHODOLOGY . 93 

B.  MULTI-ATTRIBUTE  UTILITY  THEORY . 93 

C.  VALUE  MODELING  TECHNIQUE . 94 

D.  QUANTITATIVE  VALUE  MODEL . 95 

1.  Eliciting  Utility  Curves  from  Stakeholders . 96 

2.  Percent  Traceable  Measures  Utility  Curve . 98 

3.  Days  to  Plan  Utility  Curve . 99 

4.  Quality  of  Planning  Outputs  Utility  Curve . 99 

5.  Elasticity  of  Labor  and  Elasticity  of  Duration  Utility  Curves . 100 

E.  ANALYTICAL  HIERARCHY  PROCESS . 101 

F.  RAW  DATA  VALUES . 103 

1.  Results  of  Percent  Traceable  Measures  EM . 103 

2.  Results  of  Quality  of  Planning  Outputs  EM . 105 

3.  Compiled  results  for  all  EMs . 106 

G.  CALCULATING  AN  OVERALL  UTILITY  SCORE . 106 

H.  UTILITY  SCORE  VERSUS  LIFE  CYCLE  COST  ESTIMATE . 108 

VII.  FINAL  RECOMMENDATIONS  AND  CONCLUSIONS . 11 1 

A.  PERFORMANCE  ANALYSIS . Ill 

B.  COST  ANALYSIS . 112 

C.  PERFORMANCE  VERSUS  COST  ANALYSIS . 114 

D.  CONCLUSION . 115 

E.  AREAS  FOR  FURTHER  STUDY . 116 

1.  Establish  a  Manager . 116 

2.  Initiate  Risk  Management  Strategies . 117 

3.  Modify  the  Acquisition  Process . 117 

APPENDIX  A.  PROJECT  PLAN . 119 

APPENDIX  B.  NEEDS  ANALYSIS  QUESTIONNAIRE . 133 

APPENDIX  C.  QUALITY  OF  PLANNING  OUTPUTS  QUESTIONNAIRE . 151 

APPENDIX  D.  DESIGN  ALTERNATIVES  DETAILS . 157 

APPENDIX  E.  EVALUATION  MEASURES  DETAILS . 165 

APPENDIX  F.  CORE  IDEF0  DIAGRAMS . 189 

APPENDIX  G.  EVALUATION  MEASURE:  PERCENTAGE  OF  TRACEABLE 

MEASURES . 197 

APPENDIX  H.  GLOSSARY  OF  TERMS . 209 

LIST  OF  REFERENCES . 213 

INITIAL  DISTRIBUTION  LIST . 219 


IX 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


x 


LIST  OF  FIGURES 


Figure  1.  Transitioning  Requirements  Generation  Philosophy . 3 

Figure  2.  JC3M  Systems  Engineering  Process . 10 

Figure  3.  JC3M  Problem  Refinement  Phase . 11 

Figure  4.  JC3M  Design  Alternatives  Phase . 12 

Figure  5.  JC3M  Modeling  and  Simulation  Phase . 13 

Figure  6.  JC3M  Analysis  of  Alternatives  Phase . 14 

Figure  7.  JC3M  Final  Recommendation  Phase . 15 

Figure  8.  JC3M  Functions . 22 

Figure  9.  Developing  Evaluation  Measures . 24 

Figure  10.  JC3M  Top  Level  Value  Hierarchy . 24 

Figure  11.  JC3M  Functional  Hierarchy . 25 

Figure  12.  FEDOS  Process . 31 

Figure  13.  FEDOS  Task  1  Elicit  Test  Requirements . 32 

Figure  14.  JTEM  CTM  Phases . 36 

Figure  15.  List  of  Sub  Tasks  Performed  by  SCR  to  Define  Measures . 47 

Figure  16.  SCR  Define  Measures  Subtask  Ordering . 48 

Figure  17.  JCIDS  andFCB . 50 

Figure  18.  List  of  Sub  Tasks  needed  to  Complete  Define  Measures . 51 

Figure  19.  FCB  provides  external  input  to  C4I  Test  Organization . 52 

Figure  20.  SCR  and  FCB  Variance . 53 

Figure  21.  M&S  Contribution  to  JC3M  Scoring  Matrix . 58 

Figure  22.  POW-ER  Baseline  Model . 61 

Figure  23.  Arena  Simulation  Run . 62 

Figure  24.  Example  of  Arena  Sub-Models  and  Sub-Processes . 64 

Figure  25.  Example  of  a  Process  Delay  Setting . 65 

Figure  26.  Screen  Shot  from  the  Arena  Process  Analyzer . 66 

Figure  27.  JC3M  Planning  Functions . 68 

Figure  28.  JC3M  Baseline  Data  Set  Inputs  and  Outputs . 69 

Figure  29.  Input  and  Output  of  Baseline  M&S . 71 

Figure  30.  Mapping  of  Baseline  Tasks  to  JC3M . 72 

Figure  3 1 .  Input  and  Output  of  Alternative  M&S . 73 

Figure  32.  Representative  C4I  SoS  Architecture . 75 

Figure  33.  Phases  of  JC3M  Life  Cycle . 83 

Figure  34.  Cost  Breakdown  Structure  of  JC3M . 88 

Figure  35.  High  Level  Outline  of  Value  Modeling . 94 

Figure  36.  Wymorian  Standard  Scoring  Functions  3  and  9 . 97 

Figure  37.  Utility  Function  for  Percentage  of  Traceable  Measures . 98 

Figure  38.  Utility  Function  for  Days  to  Plan  Evaluation . 99 

Figure  39.  Utility  Function  for  Quality  of  Planning  Outputs . 100 

Figure  40.  Utility  Function  for  Elasticity  of  Labor  and  Elasticity  of  Duration . 101 

Figure  41.  Sample  Analytical  Hierarchy  Process  Scale . 102 

Figure  42.  Utility  versus  Life  Cycle  Cost  Estimate  Plot . 109 


xi 


Figure  43.  Utility  versus  Life  Cycle  Cost  Estimate  Plot  Zoom . 110 

Figure  44.  Cost  Versus  Utility  with  Trendline . 115 

Figure  45.  JC3M  Team  Organization . 120 

Figure  46.  JC3M  Systems  Engineering  Process . 122 

Figure  47.  JC3M  Project  Plan . 131 

Figure  48.  Define  Problem . 167 

Figure  49.  Define  Components . 169 

Figure  50.  Define  Evaluation  Criteria . 172 

Figure  5 1 .  Ensure  Evaluation  Readiness . 176 

F igure  52 .  Adaptability . 182 

Figure  53.  Usability . 185 

Figure  54.  Repeatability . 186 

Figure  55.  FEDOS  IDEFO  Diagram . 189 

Figure  56.  MC3T  IDEFO  Diagram . 190 

Figure  57.  JTEM-CTM  IDEFO  Diagram . 191 

Figure  58.  FCB  IDEFO  Diagram . 192 

Figure  59.  FCB’s  Analyze  Measures  task . 193 

Figure  60.  SCR  IDEFO  Diagram . 194 

Figure  61.  SCR’s  Define  Measures  task . 195 

Figure  62.  Performance  Measure  Traceability . 199 

Figure  63.  Fielded  IT  &  NSS  Interoperability  Process . 200 


xii 


LIST  OF  TABLES 


Table  1.  JC3M  Critical  Evaluation  Measures . 26 

Table  2.  JC3M  Morphological  Box . 38 

Table  3.  JC3M  Alternatives . 40 

Table  4.  Reevaluated  JC3M  System  Alternatives  Datum  Design  Comparison  Matrix . 42 

Table  5.  Summary  Comparison  of  Alternatives . 54 

Table  6.  JC3M  Scoring  Matrix . 55 

Table  7.  Systems  Under  Test  during  FEDOS  2005 . 74 

Table  8.  SoS  Capabilities  Tested  during  FEDOS  ’05 . 76 

Table  9.  Elasticity  Results  for  All  Alternatives . 78 

Table  10.  POW-ER  Simulation  Results  for  JC3M  Alternatives . 79 

Table  11.  Validation  of  Labor  Hours . 80 

Table  12.  M&S  Contribution  to  JC3M  Scoring  Matrix . 80 

Table  13.  Life  Cycle  Cost  Summary . 92 

Table  14.  JC3M  Critical  Evaluation  Measures . 95 

Table  15.  AHP  Evaluation  Measure  Weights . 103 

Table  16.  Raw  Data  Matrix . 106 

Table  17.  Utility  Score  Data  Matrix . 107 

Table  18.  Quantitative  Modeling  Decision  Matrix . 108 

Table  19.  Team  Member  Listing . 121 

Table  20.  List  of  Milestones  and  Deliverables . 130 

Table  21.  Scot  Hoesly’s  Responses  to  Quality  of  Planning  Outputs  Questionnaire . 155 

Table  22.  Stephan  Bussell’s  Responses  to  Quality  of  Planning  Outputs  Questionnaire . 155 

Table  23.  Robert  Mount’s  Responses  to  Quality  of  Planning  Outputs  Questionnaire . 155 

Table  24.  Table  of  Raw  Index  Scores . 156 

Table  25.  JC3M  System  Alternatives  Datum  Design  Comparison  Matrix . 163 

Table  26.  Percent  of  Traceable  Measures  Summary  Table . 208 


xiii 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


xiv 


ACKNOWLEDGMENTS 


The  JC3M  team  would  like  to  thank  the  following  people  for  their  assistance  on 
this  project. 

Mr.  Jay  Chance,  MCTSSA,  for  his  assistance  with  process  validation. 

Mr.  Ian  Finn,  MCTSSA,  for  his  support  of  SoS  testing,  and  this  project. 

Ms.  Sau-Ping  George,  MCTSSA,  for  her  assistance  with  cost  estimation. 

Major  Timothy  McLean,  MCTSSA,  for  his  assistance  with  process  validation. 
Mr.  Gregory  Miller,  NPS,  for  his  enthusiasm  and  nuggets  of  wisdom. 

Professor  Mark  Nissen,  NPS,  for  his  assistance  with  POW-ER  simulation. 

Mr.  Linh  Nguyen,  MCTSSA,  for  his  assistance  with  systems  architecture. 

Mr.  John  Wilson,  JTEM,  for  his  assistance  with  process  validation. 


xv 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


xvi 


EXECUTIVE  SUMMARY 

Historically,  Command,  Control,  Communications,  Computers,  and  Intelligence 
(C4I)  systems  were  each  developed  to  address  a  single  warfighting  function,  whether 
used  in  a  single  Service  or  as  a  Joint  solution.  C4I  systems,  and  their  use,  have  changed 
in  response  to  advances  in  technology  as  well  as  the  growing  complexity,  scope,  and 
tempo  of  warfighting.  Individual  C4I  systems,  whether  changed  or  unmodified,  have 
been  assembled  by  developers  and  the  operating  forces  into  an  interdependent  System  of 
Systems  (SoS)  in  order  to  achieve  effects  which  a  single  system  could  not  achieve. 

The  C4I  SoS  has  grown,  evolving  in  an  almost  biological  manner  without  benefit 
of  an  overarching  architecture,  engineering  design,  or  performance  requirements. 

Without  a  SoS-level  architecture,  the  effects  of  system-level  changes  on  performance, 
capabilities,  or  standards  compliance  of  the  SoS  are  impossible  to  assess  in  an  objective, 
repeatable,  measurable,  and  effective  manner. 

Individual  systems  are  designed,  developed,  tested,  and  fielded  in  response  to 
system-level  requirements.  Individual  systems  undergo  interoperability  certification, 
which  ensures  compliance  to  standards,  but  does  not  measure  SoS  effectiveness.  Without 
measures  of  capabilities-based  performance  for  the  C4I  SoS,  system-level  changes  have 
an  unknown  effect  on  the  C4I  SoS;  the  operating  forces  can  be  saddled  with  systems  that 
do  not  work  as  assembled. 

For  example,  when  a  Forward  Observer  (FO)  sends  a  “Call  For  Fire,”  position 
location  information,  fire  control  systems,  and  communications  systems  are  used  to 
request  artillery,  rockets,  or  naval  gunfire  effects  on  a  target.  There  is  extensive  doctrine 
for  the  conduct  of  fire  support,  but  there  are  not  documented  testable  values  which  can  be 
used  to  assess  the  success  of  a  Call  for  Fire  or  the  C4I  systems  supporting  the  task.  If  a 
FO  had  to  initiate  a  Call  For  Fire  five  times,  did  the  C4I  SoS  demonstrate  a  successful 
capability  to  provide  fire  support?  What  if  the  FO  had  to  request  seven  times?  If  a 
change  to  a  component  system  required  the  FO  to  initiate  eight  times,  rather  than  seven, 
is  the  SoS  still  effective?  Without  performance  measures  for  the  C4I  SoS,  accurate 
fielding  decisions  for  SoS  components  cannot  be  made. 

xvii 


The  lack  of  performance  measures  for  the  C4I  SoS  is  a  well-known  issue,  but  the 
C4I  SoS  will  not  be  completely  rebuilt  in  order  to  include  performance  measures;  there 
are  too  many  legacy  systems,  with  significant  inertia,  that  will  continue  to  be  used. 
Architecting  the  C4I  SoS  as  a  coherent  whole  (design  from  the  top  down)  represents  an 
improbable  shift  in  philosophy.  The  “system-centric”  (bottom  up)  approach  is  likely  to 
continue. 

The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  is  in  place  to 
ensure  new  systems  are  developed  in  response  to  required  capabilities  [CJCSI  3 170.0 IF, 
2007:2],  and  that  those  systems  are  "born  joint."  However,  urgent  unfunded  needs 
statements,  urgent  warfighter  needs,  and  the  continued  "biological  growth"  of  the  SoS 
make  the  need  for  a  system  to  define  performance  measures  an  enduring  one. 

The  purpose  of  this  project  was  to  develop  the  Joint  C4I  Capability  Certification 
Measures  (JC3M)  system,  which  could  define  threshold  performance  values  for  the  C4I 
SoS.  A  disciplined  and  iterative  systems  engineering  process  was  applied  throughout  the 
project.  Stakeholders  were  identified  at  representative  C4I  test  organizations,  Subject 
Matter  Experts  (SMEs)  were  consulted,  and  their  responses  were  used  to  elicit  and 
validate  quantifiable  requirements.  A  functional  hierarchy  along  with  non-functional 
requirements,  complete  with  evaluation  measures  to  compare  the  performance  of 
alternatives,  was  created  for  JC3M,  and  validated  through  SME  consultation. 

Alternatives  to  be  considered  in  the  systems  engineering  process  consisted  of  both 
existing  systems  and  new  conceptual  systems.  A  review  of  the  literature,  combined  with 
SME  consultation,  identified  three  existing  systems  that  were  potential  JC3M  alternative 
solutions.  Two  projects  have  been  established  recently  to  address  the  lack  of 
performance  measures  for  the  C4I  SoS.  They  are  the  Joint  Test  and  Evaluation 
Methodology  (JTEM)  project’s  Capability  Test  Methodology  (CTM)  under  the  Director 
of  Operational  Test  and  Evaluation  and  the  Marine  Air  Ground  Task  Force  (MAGTF) 

C4I  Capability  Certification  Testing  (MC3T)  project  within  Marine  Corps  Systems 
Command.  A  third  existing  system,  the  Federation  Of  Systems  (FEDOS)  process  is  used 
by  MCTSSA.  Two  new  and  unique  solutions  were  created  in  order  to  add  breadth  to  the 
comparisons. 
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During  the  design  of  the  new  alternatives,  it  was  discovered  that  the  initial  goal  of 
creating  a  system  to  define  threshold  performance  values  for  the  C4I  SoS  was  not  viable. 
The  potential  number  of  configurations  of  the  C4I  SoS,  combined  with  the  number  of 
conditions  in  which  it  could  be  operated,  created  a  practically  infinite  number  of  test 
conditions.  Further,  DoD  doctrine  [CJCSM  3500. 04D,  2005,  A-3]  stipulated  that  local 
commanders,  in  response  to  the  varied  conditions  their  units  operated  in,  could  dictate 
their  own  performance  thresholds.  Any  threshold  performance  values  defined  by  a  test 
organization  might  not  be  considered  as  representative  of  user  needs,  and  thus  might  not 
be  useful.  These  discoveries  caused  the  reevaluation  of  requirements,  and  an  associated 
redesign  of  the  new  alternatives.  Instead  of  defining  threshold  performance  values  (such 
as  “seven  attempts  to  initiate  the  call  for  fire  is  an  acceptable  value”)  the  project  would 
develop  or  select  a  system  that  would  define  what  should  be  measured,  e.g.,  “attempts  to 
initiate  the  call  for  fire.”  This  would  then  lead  to  tests  to  measure  these  values,  but  leave 
it  to  the  operators  to  decide  if  it  met  their  needs. 

Modeling  and  simulation  tools  were  selected  to  evaluate  the  performance  of  the 
alternatives.  Vitech’s  CORE,  Rockwell  Automation's  Arena  and  POW-ER  (Project, 
Organization,  Work  for  Edge  Research)  developed  through  the  Virtual  Design  Team 
research  at  Stanford  University,  were  used  to  simulate  the  operation  of  the  five 
alternatives.  CORE  modeled  both  function  and  data  flows;  Arena  quantified  timing  and 
assessed  the  effects  of  varying  inputs;  and  POW-ER  assessed  processes  by  quantifying 
direct  work,  rework,  coordination  tasks,  decision  wait  time,  and  worker  backlog.  The  use 
of  these  simulation  tools  to  complement  and  calibrate  each  other  in  support  of  systems 
engineering  is  a  unique  approach  not  found  in  the  existing  literature.  Performance 
models  were  created  and  validated  against  historical  data  for  one  of  the  alternatives.  All 
alternatives  were  modeled  based  on  the  validated  model.  This  provided  a  high  level  of 
confidence  in  their  projected  performance  values. 

A  life  cycle  cost  model  was  generated  for  each  alternative.  The  historical  cost  of 
one  of  the  alternatives,  based  on  fully-burdened  labor  rates  and  actual  labor  hours,  was 
used  as  a  baseline.  A  Life  Cycle  Cost  Estimate  (LCCE)  was  created  for  each  alternative, 
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which  allowed  comparison  of  overall  cost,  as  well  development,  operating,  and 
retirement  cost  comparisons. 

An  Analysis  of  Alternatives  was  conducted  to  compare  the  estimated  performance 
of  each  alternative,  and  its  associated  cost,  to  the  other  alternatives.  SME  consultation 
was  conducted  to  estimate  the  performance  of  the  alternatives  in  some  areas.  SME 
consultation  was  also  used  to  validate  the  functions  used  to  convert  raw  performance 
evaluation  scores  into  weighted  performance  scores  for  the  alternatives. 

A  comparison  of  alternatives  based  on  the  modeling  and  simulation  results 
indicated  the  JTEM  CTM  was  projected  to  have  slightly  better  performance  than  other 
considered  alternatives,  and  had  the  median  LCCE  of  the  five  alternatives. 

The  overall  recommendation  based  on  this  study  is  to  monitor  the  development  of 
the  JTEM  CTM  for  further  maturation  as  this  project  promises  significant  improvements 
in  the  overall  utility  of  C4I  SoS  evaluations.  The  JTEM  CTM  was  initiated  in  2005;  the 
cost  for  final  development  (2007  through  2009)  is  estimated  at  $3.5M,  and  will  be  borne 
by  the  JTEM  program.  The  estimated  operating  cost,  which  would  be  borne  by  C4I  test 
organizations  implementing  the  JTEM  CTM,  is  expected  to  be  significantly  lower  than 
other  alternatives  investigated.  Changes  or  uncertainties  in  performance  or  costs 
associated  with  the  JTEM  CTM  will  require  additional  analysis  to  confirm  the  expected 
benefits  of  this  approach.  Detailed  investigation  of  the  JTEM  CTM  in  its  entirety,  and 
optimization  of  personnel  resources  and  organizations  with  a  modeling  tool  such  as 
POW-ER,  should  be  pursued.  The  JTEM  CTM  should  also  be  used  to  conduct  a  C4I  SoS 
evaluation  as  soon  as  feasible  to  validate  methods  and  processes  in  a  “real  world”  event. 
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INTRODUCTION 


A.  BACKGROUND 

Commanders  must  be  able  to  utilize  Command  and  Control  (C2)  techniques  to 
manage  the  battlespace  under  their  command.  “C2  is  essentially  about  information: 
getting  it,  judging  its  value,  processing  it  into  useful  form,  acting  on  it,  and  sharing  it  with 
others”  [Joint  Pub  6-02,  2006:1-2],  The  tasks  of  gathering,  judging,  and  processing 
information  have  been  improved  through  the  creation  of  Command,  Control, 
Communication,  Computers  and  Intelligence  (C4I)  systems.  The  intent  of  C4I  systems  is 
to  provide  commanders  with  the  tools  necessary  to  increase  their  ability  to  process  data 
and  increase  the  tempo  of  warfighting. 

Over  time,  warfighting  has  grown  in  complexity,  tempo,  and  scope; 
correspondingly,  our  military  must  be  able  to  respond  with  increased  agility  across 
greater  distances.  Furthermore,  warfighting  will  continue  to  change  in  the  future, 
requiring  our  military  to  continually  adapt  and  stay  ahead  in  the  rapidly  shifting  global 
environment.  Lieutenant  Colonel  Bemd  Horn,  Canadian  Forces,  affirms  this  position, 

To  state  that  the  battlespace  of  the  future  -  the  land,  air,  sea,  space  and 
electromagnetic  realm  where  armed  conflict  will  be  conducted  within  its 
cultural,  economic,  ecological,  environmental,  political,  social  and 
technological  contexts  -  will  be  dramatically  different  from  that  of  today 
is  to  repeat  the  strikingly  obvious  [Horn,  2003:  8], 

To  combat  adversaries  of  today  and  tomorrow,  US  forces  are  joined  not  only 
across  Services  but  with  our  international  coalition  partners  to  fight  enemies  as  a  united 
front.  Joint  C4I  systems,  therefore,  are  essential  to  the  success  of  our  military  forces  now 
and  in  the  future.  The  US  Joint  Chiefs  of  Staff  maintain  “Interoperability  is  key  to  the 
joint  force  gaining  information  superiority  in  today’s  network  enabled  environment” 
[Joint  Pub  6-02,  2006: 1-8].  The  interoperability  of  C4I  systems  is  critical  to  military 
success. 

While  the  need  for  interoperable  C4I  systems  has  long  been  recognized,  these 

systems  have  been  developed  as  solutions  to  separate  problems.  The  services  historically 

developed  C4I  systems  in  isolation;  each  system  designed  to  address  a  need  within  one 
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warfighting  function.  As  the  component  C4I  systems  have  developed,  the  ability  of 
multiple  systems  to  collectively  achieve  capabilities  beyond  the  reach  of  a  single  system 
was  also  recognized.  Systems  were  connected  to  each  other,  integrated  with  each  other, 
and  assembled  in  an  interdependent  System  of  Systems  (SoS). 

The  Predator  Unmanned  Aerial  Vehicle  (UAV)  was  originally  designed  as  a  non- 
lethal  high  altitude  reconnaissance  asset  [Airforce  Technology,  2007].  The  Helicopter 
Launched  Fire  and  Forget  Missile  (Hellfire)  was  designed  in  the  1970s  as  an  air-to- 
ground  anti-armor  missile  to  be  launched  from  low-flying  Army  and  Marine  Corps 
helicopters  [Boeing,  2007].  In  2000,  the  UAV  and  the  missile,  with  separate  missions, 
requirements,  and  service  owners,  went  through  a  “shotgun  wedding”  and  became  an 
armed  reconnaissance  and  interdiction  SoS,  culminating  in  a  successful  launch  of  a 
Hellfire  from  a  Predator  in  2001  [Checkpoint,  2007],  These  two,  formerly  completely 
independent  systems,  now  are  linked;  changes  to  missile  weight  or  range,  for  example, 
must  be  considered  in  terms  of  effects  on  the  payload  and  range  of  the  aircraft. 

After-the-fact  integration  is  not  unique  to  weapon  systems;  the  C4I  SoS  also 
contains  systems  developed  for  one  purpose,  but  now  used  for  another.  Position  Location 
Reporting  System  (PLRS)  was  originally  designed  to  provide  friendly  position  location 
information  (PLI),  identification,  and  navigation  aides.  Enhanced  PLRS  (EPLRS)  is  now 
used  by  Army  and  Marine  Corps  for  digital  network  communications.  The  Marines  have 
developed  C2  On  the  move  Digital  Over-the-horizon  Relay  (CONDOR),  which  uses 
EPLRS  to  extend  battlefield  communications  via  military  satellite  communications  and 
provide  over-the-horizon  and  on-the-move  battlefield  connectivity  [Armed  Forces 
Communications  and  Electronics  Association,  2004],  Like  the  weaponization  of  the 
Predator,  CONDOR  integrated  separately  designed  systems  into  a  new  SoS,  and  changes 
to  components  (EPLRS  and  military  satellite  communication  systems)  must  be 
considered  in  terms  of  effects  on  the  CONDOR. 

CONDOR  is  an  example  of  the  growth,  in  an  almost  biological  manner,  of  C4I 

SoS;  new  technology  continues  to  be  integrated  with  existing  systems  without  an 

overarching  architecture,  engineering  design,  set  of  performance  requirements,  or 

managers.  Without  an  overarching  view  of  the  SoS,  the  effects  of  system-level  changes 
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to  performance,  capabilities,  or  standards  compliance  on  the  SoS  are  impossible  to  assess 
in  an  objective,  repeatable,  and  measurable  manner. 


Architecting  C4I  SoS  upfront  as  a  coherent  whole,  top  down  design  represents  a 
shift  in  this  philosophy,  as  seen  in  Figure  1 .  However,  it  is  improbable  that  the  “system¬ 
centric,”  bottom  up  design,  approach  is  likely  to  cease  in  the  near  future.  Capabilities- 
based  acquisition  and  integration  requires  interoperability  evaluation  and  certification 
based  on  delivering  a  war- fighter  capability  as  part  of  a  joint  SoS. 
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C4I  Acquisition 


Integrated  at 
Department 

ir 


Bottom  Up,  Stovepiped 
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Military 

Strategy 

=5 

Joint 

Vision 

ir 

Joint  Concept  of  Operations 

Z2 


Joint  Operating  Concepts 
Joint  Integrated  Architecture 

U  ~ 

Joint  Capabilities 
Joint  Requirments 
Joint  System 


Top  Down,  Born  Joint 


Figure  1.  Transitioning  Requirements  Generation  Philosophy 

This  figure  represents  the  transitioning  philosophy  from  stovepiped  acquisition  practices 
to  acquisition  of  systems  built  from  joint  capabilities  with  the  joint  fighter  in  mind  [After 
Hoivik,  2007:  5], 

The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  process 
identifies  “. .  .the  capabilities  (and  operational  performance  criteria)  required  to 
successfully  execute  missions;  the  shortfalls  in  existing  weapon  systems  to  deliver  those 
capabilities  and  the  associated  operational  risks;  and  the  possible  solution  space  for  the 
capability  shortfalls”  [CJCSI  3 170. OIF,  2007:2],  JCIDS  is  the  long  term  Department  of 
Defense  (DoD)  answer  to  developing  systems  that  are  integrated  and  interoperable  from 
the  start. 
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The  C4I  SoS,  however,  continues  to  evolve  as  urgent  unfunded  needs  statements, 
urgent  warfighter  needs,  and  technologies  are  discovered  or  developed.  These  changes 
are  implemented  as  new  or  extended  capabilities  to  the  C4I  SoS,  which  makes  the  need 
for  a  system  to  define  performance  measures  an  enduring  one. 

Congressman  Jim  Saxton  of  New  Jersey,  member  of  the  Joint  Economic  and 
House  Armed  Service  Committees,  clearly  agrees  that  problems  exist  with  the  current 
acquisition  practices. 

Presently,  the  department  (of  Defense)  allows  its  individual  military 
services,  agencies  and  field  activities  to  determine  their  own  IT 
[Information  Technology]  needs.  This  stove-piped  approach  has  led  to  the 
confusing  and  complex  C4ISR  (Command,  Control,  Communication, 
Computers,  Intelligence,  Surveillance,  and  Recognizance)  landscape  that 
exists  today.  This  environment  has  resulted  in  waste,  redundancy  and  lack 
of  interoperability  in  IT  systems  and  capabilities  for  our  warfighters 
[Saxton,  2003]. 

Today’s  Joint  C4I  SoS  are  required  to  transport  and  process  shared  data 
throughout  the  operating  forces.  The  Joint  Communication  System  doctrine  elucidates 
this  point. 

A  joint  force  that  is  linked  and  synchronized  in  time  and  purpose  is 
considered  networked.  ...  To  do  this,  the  communications  system  must  be 
interoperable,  agile,  trusted,  and  shared.  Problems  are  abundant  because 
there  is  no  baseline,  standard  configuration,  or  overall  management  of  the 
SoS  [Joint  Pub  6-02,  2006:  viii] . 

It  is  this  lack  of  baselines,  standard  configurations,  and  overall  management  of 
SoS  that  make  defining  C4I  SoS  capability  performance  measures  so  difficult.  Without 
performance  measures  for  the  C4I  SoS,  however,  accurate  fielding  decisions  for  SoS 
components  cannot  be  made.  Some  specific  examples  of  the  negative  consequences  of 
system-focused  fielding  decisions  on  SoS  capabilities  are  provided  in  Chapter  I,  Section 
C. 


C4I  system-level  acquisition,  testing,  and  management  are  well  understood,  and 
individual  systems  have  documented  performance  requirements  (e.g.,  Capabilities 
Development  Documents  or  Operational  Requirements  Documents).  Processes  and 
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methods  for  designing  and  executing  C4I  system  tests  are  well  defined  and  executed,  but 
testing  methods  at  the  SoS  level  are  not  well  defined,  nor  are  consistent  standards  and 
practices  applied.  A  complicating  factor  is  that  real  instances  of  the  C4I  SoS  have  a 
practically  infinite  number  of  possible  configurations. 

C4I  SoS  have  ever-changing  configurations  that  do  not  have  formally  established 
performance  requirements,  nor  standard  SoS  capabilities  that  can  be  used  to  generate 
performance  evaluation  measures.  There  is  not  a  clear  understanding  of  how  to  manage 
or  assess  C4I  SoS  performance  or  C4I  SoS  capability  to  support  joint  or  single  Service 
missions. 

The  Joint  Interoperability  Test  Command  (JITC)  is  the  sole  DoD  certification 
authority  for  joint  and  combined  interoperability  [CJCSI  6212. 01D,  2007:  5],  JITC  tests 
the  interoperability  of  systems,  but  this  only  proves  that  system  interfaces  function. 

There  is  no  agency  that  assesses  the  capability  of  a  SoS  to  accomplish  a  task  that  requires 
the  coordinated,  successful  integration  of  functions  and  interfaces  across  multiple 
systems.  Indeed,  JCIDS  defines  interoperability  to  include  “both  the  technical  exchange 
of  information  and  the  end-to-end  operational  effectiveness  of  that  exchanged 
information  as  required  for  mission  accomplishment”  [CJCSI  3 170. OIF,  2007]. 

The  goal  of  this  project  was  to  design  a  Joint  C4I  Capability  Certification 
Measures  (JC3M)  system  to  fill  the  current  capability  gap  that  exists  in  the  determination 
of  a  system’s  effect  on  SoS  performance.  JC3M  is  important  because  it  is  intended  to 
provide  SoS  users,  as  well  as  the  acquisition  community,  with  much-needed  performance 
requirements  for  the  design  of  new  systems,  integration  of  legacy  systems,  and  SoS 
testing.  JC3M  is  also  important  because  it  will  provide  system  integrators  a  tool  to  assess 
integration  formally,  it  will  document  system  capabilities  and  construction,  and  it  will 
provide  confidence  to  the  warfighter  that  the  C4I  SoS  works.  C4I  SoS  have  been  custom 
built  to  date,  with  all  the  Configuration  Management  (CM),  troubleshooting,  training,  and 
support  challenges  this  “one-off’  approach  implies.  With  a  consistent  assessment 
methodology  and  a  documented  baseline  configuration,  C4I  SoS  evaluation  becomes 
repeatable.  This  repeatability  allows  CM,  training,  troubleshooting,  and  knowledge 
management  costs  to  shrink. 
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The  JC3M  team  defined  a  system  for  use  by  C4I  test  organizations  that  will 
identify  the  desired  war  fighting  capabilities  of  the  C4I  SoS  and  ensure  that  the  system 
under  test  (SUT)  meets  these  requirements.  The  JC3M  system  included  Doctrine, 
Organization,  Training,  Materiel,  Leadership,  Personnel,  and  Facilities  (DOTMLPF) 
considerations  in  designing  the  system  to  be  used  by  an  organization,  following 
repeatable  processes,  and  using  consistent  resources. 

Parallel  efforts  are  underway  to  address  SoS  evaluation,  and  these  approaches 
were  considered  along  with  other  candidate  solutions  for  JC3M.  Existing  efforts  include 
the  JTEM  CTM,  which  is  addressing  Joint  SoS  performance  evaluation  from  the  Office 
of  Secretary  of  Defense  (OSD)  level.  Marine  Corps  Systems  Command 
(MARCORSYSCOM),  the  acquisition  organization  for  the  Marine  Corps,  is  approaching 
the  issue  from  a  Service  perspective.  MARCORSYSCOM  has  tasked  Marine  Corps 
Tactical  Systems  Support  Activity  (MCTSSA)  to  develop  MC3T,  a  methodology  for 
managing  the  MAGTF  C4I  SoS  as  a  single  system,  in  accord  with  modern  systems 
engineering  practices.  MC3T  will  manage  the  MAGTF  C4I  SoS  as  a  set  of  SoS-level 
capabilities,  rather  than  as  a  fixed  hardware  or  software  baseline. 

JC3M  leveraged  MC3T  and  JTEM  efforts,  cooperating  and  consulting  with  both 
organizations  as  stakeholders.  Individual  JC3M  team  members  will  work  with  both 
JTEM  and  MC3T  as  their  processes  mature  and  are  implemented.  JTEM  is  expected  to 
complete  development  in  2009;  MC3T  is  conducting  a  proof  of  concept  event  from 
October  through  December  2007. 

B.  PROBLEM  STATEMENT 

Information  Age  C4I  systems  have  grown  biologically  into  an  interdependent 
entity,  a  System  of  Systems.  There  is  no  clear  consensus  on  how  to  evaluate  a  C4I  SoS 
capability,  that  is,  a  task  only  achievable  by  interdependent  multiple  components.  Current 
inter-operability  certification  processes  only  partially  address  SoS  end-to-end  evaluation; 
systems  are  certified  and  delivered  with  interfaces  functioning  from  a  technical 
perspective  but  with  less  than  optimum  results  or  end-user  dissatisfaction  from  an 
operational  effectiveness  standpoint.  Capabilities-based  acquisition  requires 
interoperability  evaluation  and  certification  based  on  delivering  a  war- fighter  capability 
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in  an  operational  environment  as  part  of  the  SoS.  The  primitive  need,  as  identified  by  the 
MC3T  team  lead,  was  for  testable  threshold  values  to  use  in  the  design  and  conduct  of  a 
C4I  SoS  test.  This  statement  is  based  on  experience  at  MCTSSA  in  conducting  SoS  tests 
for  acquisition  agencies.  This  led  the  team  to  the  problem  statement  below: 

There  is  no  system  that  defines  and  compares  System  of  Systems 

performance  measures  to  warfighter  needs  in  an  objective  and  measurable 

way  [Finn,  2007], 

C.  OPERATING  FORCES  NEED  FOR  JC3M 

The  lack  of  SoS-level  performance  measures  is  not  an  issue  that  only  affects  the 
test  community;  there  are  consequences  to  the  operating  forces  as  well.  Two  examples  of 
system-level  changes  which  were  tested  and  fielded  without  SoS-level  performance 
measurements  are  provided.  The  test  cases  are  summarized  below. 

1.  Example  1  -  Voice  Switch  Fielding 

The  first  example,  testing  and  fielding  of  voice  switches,  affected  operating  forces 
communication  networks.  This  problem  involves  three  different  voice  switches  fielded 
over  eighteen  years  for  the  Marine  Corps.  A  voice  switch  is  a  system  of  electronic 
components  that  connects  telephone  calls,  providing  voice  communications  between 
calling  and  called  subscribers.  When  placing  a  telephone  call,  the  person  initiating  the 
communications  is  a  calling  subscriber  and  person  receiving  the  communications  is  a 
called  subscriber.  When  a  calling  subscriber  makes  a  phone  call  to  a  called  subscriber,  the 
calling  subscriber  hears  a  ring  back  tone  (ring  . . .  ring,  in  the  ear  piece)  and  the  called 
subscriber’s  phone  physically  rings.  The  called  subscriber  picks  up  the  phone  and  the 
voice  communication  path  is  established.  This  communication  capability  is  made 
available  through  the  use  of  voice  switches. 

The  Unit  Level  Circuit  Switch  (ULCS),  Integrated  Serviced  Digital  Network 
(ISDN)  Gateway  Exchange  (IGX),  and  Compact  Digital  Switch  (CDS)  are  switches 
interoperated  through  trunks  which  provide  voice  paths  between  two  switches.  Three 
different  types  of  telephones  were  involved:  Secure  Terminal  Equipment  (STE)  in  Plain 
Old  Telephone  System  (POTS)  mode  for  non  secure  calls,  Digital  Secure  Voice  Terminal 
(DSVT)  for  secure  calls  and  Digital  Non-secure  Voice  Terminal  (DNVT)  for  non  secure 
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calls.  The  STE  was  connected  to  IGX;  the  DSVT  and  DNVT  phones  were  connected  to 
the  CDS  and  ULCS 

Each  voice  switch  is  similar,  and  provides  secure  and  non  secure  voice 
communication  capabilities.  CDS  and  ULCS  switches  are  designed  and  built  to  the  tri¬ 
service  tactical  (TRITAC)  communication  specifications.  The  IGX  was  required  to 
interface  with  both  the  CDS  and  ULCS  by  implementing  a  specification  the  same  as  the 
TRITAC  specification.  The  architecture  can  be  configured  in  many  ways  forming  the 
SoS  that  provides  a  voice  communication  capability 

Testing  was  conducted  and  a  voice  call  was  successfully  completed  between  IGX 
and  CDS  using  a  non  secure  telephone.  The  conclusion  was  that  the  IGX  and  CDS 
worked  according  to  the  system-level  specifications.  Voice  calls  were  successfully 
completed  between  CDS  and  ULCS  using  all  three  different  telephones.  The  conclusion 
was  that  CDS  and  ULCS  worked  according  to  the  system-level  specifications.  After 
testing,  the  systems  were  fielded. 

After  fielding,  the  operating  forces  implemented  an  architecture  using  all  three 
switches  connected  in  series;  the  IGX  was  connected  to  the  CDS,  which  was  in  turn 
connected  to  the  ULCS.  Users  found  a  call  could  not  be  successfully  completed  from  a 
non  secure  phone  from  the  IGX  through  the  CDS  to  a  DNVT  called  subscriber  at  ULCS 
switch.  The  called  subscriber  DNVT  rang  but  the  calling  subscriber’s  non  secure  phone 
did  not  get  a  ring  back  tone.  The  interoperability  and  specifications  had  been  tested,  and 
all  three  systems  passed.  The  problem  was  that  the  architecture  that  supported  the 
capability,  as  implemented  by  the  user,  was  not  tested.  As  a  result,  these  switches  were 
fielded,  and  it  was  only  after  delivery  to  the  operating  forces  that  this  failure  was 
observed.  This  failure  continues  to  this  day,  forcing  workarounds  for  communicators, 
and  limiting  their  use  of  voice  switches  in  all  configurations. 

The  JC3M  system  would  have  identified  the  SoS  architecture(s)  that  provide  the 
capability,  including  viable  configurations,  tested  those  configurations,  and  ensured  the 
configuration  was  documented  for  use  by  the  operating  forces.  Without  JC3M,  however, 
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these  switches  were  fielded  and  the  operating  forces  discovered  after  fielding  specific 
configurations  did  not  work. 

2.  Example  2  -  Gateway  Exchange  Interoperability 

The  second  example,  gateway  exchange  interoperability,  also  affected  operating 
forces  communication  networks.  The  ISDN  IGX  and  High  Density  Exchange  (HDX)  are 
voice  switches  with  a  Radio  Wireline  Interface  (RWI)  Circuit  Card  Assembly  (CCA)  that 
provides  interface  capability  with  radios.  Single  Channel  Ground  to  Air  Radio  System 
(SINCGARS)  radios  provide  wireless  voice  communications.  IGX  ,  HDX  and 
SINCGARS  are  assembled  as  a  SoS  to  provide  voice  communications  capability  to 
subscribers  through  voice  switches  and  radio  operators. 

There  were  three  different  versions  of  software  used  in  the  IGX  and  HDX,  and 
testing  was  performed  on  all  three  versions.  The  configuration  of  the  SINCGARS  and 
RWI  CCA  did  not  change. 

The  version  interoperability  between  IGX  version  6.0a  and  SINCGARS  radio  was 
successful,  but  IGX  version  6.1a  and  HDX  version  1.0a  were  not  successful.  When 
versions  6.1a  and  HDX  1.0a  were  developed,  they  were  not  tested  with  the  RWI  CCA. 

The  JC3M  system  would  have  identified  the  SoS  architecture(s)  that  provides  the 
capability,  to  include  the  legacy  system  hardware,  the  RWI  CCA,  with  HDX.  During 
SoS  testing,  the  interoperability  problem  between  voice  switches  and  SINCGARS  radios 
would  have  been  identified  and  fixed,  before  it  was  fielded.  Although  the  RWI  CCA  is  at 
the  end  of  its  life  cycle,  it  is  part  of  HDX,  the  latest  voice  switch.  Had  there  been  a  JC3M 
system,  there  would  have  been  a  documented  requirement  for  RWI  interface  with  IGX 
and  HDX  prior  to  testing 

D.  THE  SYSTEMS  ENGINEERING  DESIGN  PROCESS 

1.  Overview 

The  team  implemented  a  Systems  Engineering  (SE)  approach  that  started  with  the 
identification  of  customers’  needs  and  proceeds  through  the  phases  illustrated  in  Figure  2 
until  a  recommended  solution  was  generated.  Each  phase  was  broken  down  into  one  or 
more  subtasks  that  break  the  larger  phases  into  elements.  At  the  end  of  each  phase  the 
opportunity  to  re-evaluate  progress  and  return  to  a  prior  stage  for  refinements  was 
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available  if  necessary.  This  refinement  process  was  critical  to  adapting  the  project  to 
ensure  the  customers’  and  stakeholders’  needs  are  achieved.  The  SE  process  model 
below  was  based  on  a  combination  of  System  Engineering  Design  Process  (SEDP)  [Paulo 
(a),  2005]  and  State  the  problem,  Investigate  alternatives,  Model  the  system,  Integrate, 
Launch  the  system,  Assess  performance,  and  Re-evaluate  (SIMILAR),  modified  and 
combined  to  fit  this  project. 

The  SE  process  presented  in  Figure  2  is  slightly  different  than  the  original  SE 
process  provided  within  the  Project  Management  Plan  (PMP)  (Appendix  A).  In  the 
original  SE  process,  the  Modeling  phase  composed  a  single  phase  and  the  Simulation  and 
Analysis  of  Alternatives  (AoA)  elements  were  joined  in  the  next  phase.  Through  the 
implementation  of  the  SE  process,  the  team  realized  that  this  did  not  truly  represent  the 
phases  as  they  were  logically  and  physically  executed.  Therefore,  the  simulation  element 
was  moved  to  the  modeling  phase.  The  result  is  a  phase  boundary  change,  and  not  a 
reorganization  of  the  SE  process. 


Figure  2.  JC3M  Systems  Engineering  Process. 

This  SE  process  model  is  based  on  combination  of  SEDP  by  Professor  Paulo  and 
SIMILAR  System  Engineering  Process  Model  from  INCOSE.  Central  to  the  JC3M 
Systems  Engineering  Process  is  the  ability  to  re-evaluate  after  each  element  to  update  or 
rework  an  element  as  required  prior  to  moving  on. 
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2.  JC3M  Systems  Engineering  Process  Phases 

Each  of  the  JC3M  Systems  Engineering  process  phases  is  briefly  described  to 
provide  a  summary  of  the  SE  process  the  team  utilized  throughout  this  project.  Each 
phase  and  the  associated  elements  are  described  in  greater  detail  within  Appendix  A. 
a.  Problem  Refinement  Phase 

The  team  utilized  the  Problem  Refinement  phase  to  clarify  stakeholders’ 
needs  and  begin  managing  the  JC3M  project  risks.  The  outputs  of  this  phase  allowed  the 
team  to  design  multiple  solutions  for  the  JC3M  system  problem.  The  Problem 


Refinement  Phase  is  composed  of  three  key  elements:  Needs  Analysis,  Requirements 
Generation  and  Analysis,  and  Values  System  Design,  which  includes  definition  of 
Evaluation  Measures  (EM).  Each  element  and  the  associated  inputs  and  outputs  are 
provided  in  Figure  3.  It  was  also  during  the  Problem  Refinement  phase  that  the  team 


initialized  risk  management  techniques. 


Problem  Refinement 


Need  Analysis 

Input: 

•  Initial  Problem 
«  Statement, 

•  Stakeholders’ 
Needs/Wants 

Output: 

•  Revised  Problem 
Statement 


Requirement 
Generation  & 
Analysis 

Input: 

*  Revised  Problem 
Statement 

Output: 

*  Initial  Problem 
Refinement  Report 
(PRR) 

*  Functional  Hierarchy 


Value  System 
Design 

Input: 

■  Initial  PRR 
Output: 

-  EM 

■  Value  Hierarchy 
Tree 


Figure  3.  JC3M  Problem  Refinement  Phase. 

The  Problem  Refinement  Phase  consists  of  three  key  elements,  each  providing  outputs 
for  later  phases  and  elements.  Central  to  the  JC3M  Systems  Engineering  Process  is  the 
ability  to  re-evaluate  after  each  element  to  update  or  rework  an  element  as  required  prior 
to  moving  on. 
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b.  Design  Alternatives  Phase 

The  team  utilized  the  Design  Alternatives  phase  to  generate  multiple 
candidate  solutions  to  the  problem,  establish  feasibility  criteria  and  apply  those  criteria  to 
eliminate  those  alternatives  that  were  clearly  infeasible.  The  creation  of  these  solutions 
required  the  outputs  of  the  Problem  Refinement  phase.  The  outputs  of  this  phase  were 


used  for  the  creation  of  models  in  the  Modeling  and  Simulation  phase.  The  Design 
Alternatives  Phase  is  composed  of  the  three  key  elements:  Alternatives  Generation, 
Feasibility  Criteria,  and  Alternative  Scoring.  Each  element  and  the  associated  inputs  and 


outputs  are  provided  in  Figure  4. 


Design  Alternatives 


Figure  4.  JC3M  Design  Alternatives  Phase. 

The  Design  Alternatives  phase  consists  of  three  key  elements,  each  providing  outputs  for 
later  phases  and  elements. 
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c.  Modeling  and  Simulation  Phase 

The  team  utilized  the  Modeling  and  Simulation  phase  to  generate  models 
based  on  the  alternatives  selected  in  the  Design  Alternatives  phase.  Once  the  models 
were  built,  the  alternatives  were  simulated  to  provide  results  for  comparison  in  the  AoA 
phase.  The  Modeling  and  Simulation  phase  is  composed  of  the  two  key  elements:  Model 
Development  and  Simulation.  Each  element  and  the  associated  inputs  and  outputs  are 


provided  in  Figure  5. 


Modeling  and  Simulation 


Model  Development 

Inputs: 

*  Feasible  Alternatives 

*  Joint  Service  Architectures 

*  Functional  Hierarchy 
Output: 

*  Detailed  Functional  and 
Behavioral  Models 


Simulation 

Inputs: 

*  Detailed  Functional  & 
Behavioral  Models 

Outputs: 

*  Simulation  Results 

*  Cost  Data 


Figure  5.  JC3M  Modeling  and  Simulation  Phase. 

The  Modeling  and  Simulation  phase  consists  of  two  key  elements,  each  providing  outputs 
for  later  phases  and  elements. 
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d.  Analysis  of  Alternatives  Phase 

The  team  utilized  the  Analysis  of  Alternative  phase  to  examine  each 


alternative’s  cost  in  light  of  its  performance  that  resulted  from  the  Modeling  and 
Simulation  phase.  Multi-attribute  value  modeling  techniques  were  employed  to  evaluate 
and  rank  each  of  the  alternatives.  The  output  of  this  phase  provided  utility  scores  and  life 
cycle  cost  estimates  for  each  alternative  to  the  Final  Recommendation  phase  for  decision 
making.  The  Analysis  of  Alternatives  Phase  is  composed  of  the  two  key  elements:  Cost 
Analysis  and  Multi-attribute  Value  Modeling.  Each  element  and  the  associated  inputs 
and  outputs  are  provided  in  Figure  6. 


Analysis  of  Alternatives 


Inputs: 

•  Detailed  Functional  and 


Multi-Attribute 
Value  Modeling 


Behavioral  Models 

•  Cost  Data 
Output: 

•  SoS  Life  Cycle  Cost 


Inputs: 

*  Simulation  Results 

*  Scoring  Criteria 

*  Quantitative  Value 


Estimate 


Modeling  Matrix 

•  EM  Weights 

•  Value  Functions 
Outputs: 

•  Total  Utility  For  Each 


Figure  6.  JC3M  Analysis  of  Alternatives  Phase. 

The  Analysis  of  Alternatives  phase  consists  of  two  key  elements,  each  providing  outputs 
for  later  phases  and  elements. 
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e.  Final  Recommendation  Phase 

The  team  utilized  the  Final  Recommendation  phase  to  assemble  all  earlier 
inputs,  leading  to  a  preferred  alternative  and  a  cohesive  recommendation  for 
implementation,  which  is  published  in  this  final  report.  The  output  of  the  Final 
Recommendation  Phase  will  be  provided  to  both  the  JTEM  and  MC3T  teams  for  their 
use.  The  MC3T  implementation  team  is  a  critical  stakeholder,  but  their  schedule  includes 
a  proof  of  concept  event  in  October  2007.  Based  on  this  schedule,  the  MC3T  team 
concluded  they  could  not  wait  for  JC3M  project  outputs.  MC3T  implemented  an  interim 
process,  and  will  review  the  JC3M  project  report  for  inclusion  of  applicable 
recommendations  as  MC3T  refines  their  processes.  The  Final  Recommendation  Phase  is 
composed  of  one  key  element:  Recommendation.  The  element  and  the  associated  inputs 
and  outputs  are  provided  in  Figure  7. 

Final  Recommendation 


Recommendation 

Inputs: 

*  Utility  Vs.  Life  Cycle  Cost 
Estimate 

Output: 

•  Stakeholder 
Recommendation 


Figure  7.  JC3M  Final  Recommendation  Phase. 

The  Final  Recommendation  phase  consists  of  one  key  element.  This  phase  is  the 
culmination  of  all  the  prior  phases  and  provides  the  recommendation  to  the  stakeholders 
for  concurrence. 
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f.  Assess  Performance  Phase 

This  project  does  not  completely  conclude  after  delivery  of  the  final 
report.  Some  members  of  the  team  will  continue  to  work  with  MC3T,  ITEM,  and  other 
stakeholders  after  this  project  concludes.  Some  members  of  the  team  already  participate 
in  the  JTEM  Capability  Testing  Community  of  Interest,  where  they  will  share  their 
experiences  with  other  members  and  continue  to  advance  the  art  of  capability  testing. 


16 


II.  PROBLEM  REFINEMENT 


Problem  Refinement  was  the  stage  of  the  JC3M  process  that  refined  the  initial 
understanding  of  the  problem  into  a  workable  system  design.  The  JC3M  SE  process, 
discussed  in  Chapter  I,  was  used  in  an  iterative  manner  both  to  direct  the  process  of 
developing  a  solution,  as  well  as  to  guide  the  team  away  from  immediately  designing  the 
JC3M  solution  before  all  the  requirements  were  understood.  Each  stage  of  the  SE 
process  was  iterative;  each  phase  and  element  presented  the  opportunity  to  re-evaluate 
progress  and  return  to  a  prior  stage  for  refinement  if  necessary.  While  this  presented  a 
challenge  in  managing  the  growth  of  the  project  (“scope  creep”),  it  was  also  critical  to 
adapting  the  project  to  ensure  the  needs  of  the  stakeholders  as  identified  in  this  Chapter 
were  met. 

To  accomplish  the  transformation  from  perceived  need  to  documented 
requirements,  the  SEDP  [Paulo  (a),  2005]  calls  for  Stakeholder  Analysis,  Input-Output 
Modeling,  and  Functional  Decomposition.  The  JC3M  team  blended  the  SEDP 
methodology  with  a  modified  customer  requirements  capture  process  [Stevens  et  al., 
1998:  28]  and  separately  collected  user  and  system  requirements,  organized  the 
requirements,  and  created  requirements  documentation. 

Problem  Refinement  consisted  of  three  tasks:  Needs  Analysis,  Requirement 
Generation  and  Analysis,  and  Value  System  Design.  Needs  Analysis  task  identified 
stakeholders  relevant  to  the  problem.  While  some  [Whitten  and  Bentley,  2007:7;  Stevens 
et  al,  1998:21]  define  stakeholders  as  only  persons,  the  JC3M  team  chose  to  include 
selected  organizations  as  stakeholders  because  the  duration  of  organizations  and  the 
missions  those  organizations  perform  outlast  individuals.  MCTSSA  has  been  testing  C2 
software  since  1972,  yet  none  of  the  original  MCTSSA  personnel  remain  at  the 
organization. 

The  Needs  Analysis  portion  of  the  SEDP  conducted  an  investigation  of  the 
problem,  in  order  to  refine  the  problem  from  a  primitive  need  to  an  effective  need,  and 
deliver  a  refined  problem  statement.  Needs  Analysis  also  served  as  a  control  gate,  which 
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rendered  a  go  or  no-go  decision  for  continuation  of  the  design  process.  If  the  Revised 
Problem  Statement,  the  primary  output  of  Needs  Analysis,  had  revealed  the  effective 
need  was  already  met,  thus  stopping  the  design  process,  the  JC3M  team  would  have 
undertaken  a  different  project.  Because  the  Refined  Problem  Statement  showed  an 
effective  need  existed,  it  provided  justification  to  continue  further  with  the  design 
process. 

Requirements  Generation  and  Analysis  took  the  Refined  Problem  Statement  and 
performed  system  decomposition  to  clarify  requirements.  Those  requirements  were 
decomposed  into  critical  functions  and  functional  flows.  The  outputs  of  Requirements 
Generation  and  Analysis  were  the  Problem  Refinement  Report  and  the  JC3M  Functional 
and  Value  Hierarchies. 

Value  System  Design  took  the  Problem  Refinement  Report  and  Functional 
Hierarchy  and  identified  characteristics  and  measurable  parameters  of  the  JC3M  system, 
which  were  used  later  to  evaluate  alternative  solutions.  Outputs  of  Value  System  Design 
were  Evaluation  Measures  (EM)  as  well  as  the  JC3M  values  and  functions,  documented 
in  the  JC3M  Value  Hierarchy 
A.  NEEDS  ANALYSIS 

Needs  Analysis  consists  of  identifying  stakeholders  and  their  requirements 
[Verma,  2006],  The  functional  need  identified  by  stakeholders  was  translated  into 
qualitative  and  quantitative  requirements  later  in  the  JC3M  design  process. 

1.  Stakeholder  Identification 

Key  stakeholders  must  be  identified  in  order  to  determine  their  requirements. 
Whitten  and  Bentley  [2007]  and  Stevens  [1998]  described  the  need  for  stakeholder  input, 
but  assumed  stakeholders  were  identified.  Verma  [2006:  8]  stipulates  that  stakeholders 
can  be  active  or  passive,  yet  does  not  explain  how  internal  or  external  stakeholders  can  be 
identified.  Identification  of  the  people  and  organizations  that  were  to  become  JC3M 
stakeholders  was  critical,  so  the  team  used  several  methods  in  this  stage. 

First,  the  team  created  a  list  of  potential  stakeholders.  An  internal  review  of  the 
team  confirmed  a  substantial  experience  base  in  C4I  SoS  testing  in  general,  current  test 

processes,  and  the  development  of  new  test  processes.  Team  members  were  experienced 
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in  hardware  and  systems  testing;  systems  development  and  testing;  C4I  SoS  testing;  and 
the  development  of  new  test  methodologies.  This  experience  base  was  polled,  and  a  list 
of  potential  stakeholder  organizations  was  created: 

■  Joint  Interoperability  Test  Command,  Ft.  Huachuca,  AZ 

■  Marine  Corps  Operational  Test  and  Evaluation  Activity  (MCOTEA), 
Quantico,  VA 

■  U.S.  Army  Test  and  Evaluation  Command,  Alexandria,  VA 

■  U.S.  Navy  Operational  Test  and  Evaluation  Force,  Norfolk,  VA 

■  U.S.  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  AFB, 

NM 

■  MCTSSA,  Camp  Pendleton,  CA 

■  U.S.  Army  Program  Executive  Office  for  Command,  Control,  and 
Communications  Tactical  (PEO  C3T) 

As  the  project  continued,  some  members  were  assigned,  as  part  of  their  normal, 
non-academic  duties,  to  investigate  and  develop  a  new  test  methodology  (see  Chapter  III, 
Section  A.l.  Known  Alternatives  -  MC3T).  As  part  of  this  effort,  external  to  JC3M,  the 
team  was  introduced  to  the  JTEM  Capability  Test  Methodology  (CTM)  process.  The 
JTEM  CTM  team  proved  to  be  a  very  valuable  participant  in  defining  requirements, 
validating  key  JC3M  concepts  and  service,  and  an  alternative  JC3M  solution  (see  Chapter 
III  section  A.l  Known  Alternatives  -  JTEM  CTM  for  details). 

The  team  refined  the  list  of  potential  stakeholders,  targeting  organizations  that 
were  interested,  willing  to  participate,  and  representative  of  a  portion  of  the  C4I  test 
community.  The  team  identified  organizations  that  provide  service-specific  test 
perspective  on  current  test  challenges  (MCTSSA,  PEO  C3T);  DoD-level  perspective  on 
current  and  future  test  developments  (JITC,  JTEM  CTM);  and  academic  and  theoretical 
perspective  (NPS).  The  key  stakeholders  that  were  consulted  throughout  the  duration  of 
the  project  were:  JITC,  MCTSSA,  JTEM  CTM,  PEO  C3T,  and  NPS. 

2.  Stakeholder  Initial  Requirements 

As  stakeholders  were  identified,  perceived  needs  were  reviewed  and  documented 
in  an  initial  problem  statement.  The  primitive  need  was  for  testable  threshold  values  to 
be  used  in  the  design  and  conduct  of  a  C4I  SoS  test.  While  Finn  [2007]  defined  what  he 
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perceived  as  the  effective  need,  a  system  to  evaluate  SoS  performance,  Bjorkman 
[(a), 2007]  agreed  with  the  team  that  the  definition  of  test  criteria  is  a  central  challenge  in 
SoS  testing.  SoS  complexity  and  test  challenges  are  not  limited  to  DoD,  but  in  the 
interests  of  managing  the  scope  of  the  project,  the  JC3M  team  will  focus  on  DoD  C4I 
SoS. 

B.  REQUIREMENTS  GENERATION  AND  ANALYSIS 

Requirement  Generation  and  Analysis  was  another  critical  stage  in  the  design 
process,  because  it  further  clarified  requirements,  identified  critical  functions  of  the 
system,  and  documented  those  requirements  for  use  in  all  later  stages. 

1.  Requirements  Identification 

The  goal  of  requirements  identification  was  to  move  from  the  primitive  need 
expressed  above  to  a  more  refined,  more  accurate  effective  need.  The  primitive  need  is 
reflective  of  the  stakeholder’s  bias,  experience,  and  perceptions  [Paulo  (c),  2005:1]  so  the 
JC3M  team  explored  this  perceived  need,  and  looked  at  the  larger  context  of  the  problem, 
to  find  the  true,  underlying  (effective)  need. 

The  transition  from  primitive  need  to  effective  need  was  based  on  stakeholder 
inputs  that  were  captured  via  stakeholder  focused  interviews  and  were  aided  by 
questionnaires.  After  reviewing  the  primitive  need,  the  JC3M  team  generated  a 
questionnaire  to  solicit  comments,  perspective,  and  specific  guidance  from  potential 
stakeholders.  The  questionnaire  was  reviewed  verbally  or  provided  for  review 
electronically.  The  questionnaire  and  responses  are  provided  as  Appendix  B. 
Respondents  include: 

■  JTEM:  verbally  reviewed  with  Dr.  Dave  Dryer  and  Mr.  Tim  Beach  on  14 
February  2007 

■  MC3T:  reviewed  in  person  with  Mr.  Ian  Finn  on  23  February  2007. 

■  Central  Technical  Support  Facility  (CTSF):  provided  in  February  2007. 
Comments  not  released. 

■  JITC:  questionnaire  provided  in  February  2007.  Comments  received  28 
Feb  2007 
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The  initial  version  was  formulated  to  capture  SoS  information,  but  did  not  capture 
information  necessary  to  refine  JC3M  requirements.  The  revised  questionnaire  was 
submitted  to  JITC,  and  their  response  is  also  included  in  Appendix  B.  In  their  response, 
JITC  Subject  Matter  Experts  (SMEs)  provided  detailed  guidance  on  overarching 
interoperability  and  integration  requirements,  DoD  mandated  events  and  processes,  and 
their  role  in  standards  conformance  and  interoperability  testing.  This  information  was 
central  to  the  development  of  the  two  new  alternative  solutions  for  JC3M.  The  JITC 
responses  also  cemented  the  developing  relationship  between  JITC  and  the  team.  JITC 
SMEs  were  consulted  throughout  the  project,  participated  in  both  of  the  project  formal 
reviews,  and  assisted  with  the  validation  of  requirements  and  approaches. 

2.  Requirements  Analysis 

Neither  interviews  with  stakeholders  nor  internal  team  reviews  could  identify  a 
system  to  satisfy  the  primitive  need  as  articulated  by  Finn  [2007].  The  team  determined 
the  effective  need  was  for  a  system  that  could  define  threshold  performance  values  for  the 
C4I  SoS.  These  threshold  performance  values  could  be  used,  in  turn,  by  existing  SoS  test 
systems  to  evaluate  SoS  performance.  Internal  and  external  reviews,  as  well  as  a  search 
of  the  existing  literature,  showed  there  was  not  a  system  in  place  to  meet  the  effective 
need. 

As  a  solution  to  this  problem,  the  JC3M  system  was  to  be  designed  to  meet  the 
effective  need  in  the  planning,  conduct,  and  reporting  of  C4I  SoS  performance.  Note  that 
existing  Joint  or  Service  valid  processes  were  used  where  applicable;  the  JC3M  system 
does  not,  for  example,  define  how  to  conduct  a  stakeholder  meeting,  define  the  creation 
of  test  scripts,  or  define  how  to  communicate  test  results  to  stakeholders.  Because  the 
conduct  of  assessments  is  well-understood  and  executed  by  C4I  test  organizations,  as  is 
reporting  the  results  of  those  assessments,  the  JC3M  system  is  focused  on  the  steps 
leading  up  to  and  including  planning  of  an  assessment. 

The  functional  analysis  of  JC3M  requirements  was  conducted  by  the  JC3M  team, 
and  the  model  was  validated  in  discussions  with  Mr.  Ian  Finn,  MC3T  test  lead  at 
MCTSSA,  as  well  as  at  JC3M  Program  Reviews  (16  March  2007,  08  June  2007). 
Attendees  included  representatives  from  JITC,  MCTSSA,  PEO  C3T,  and  JTEM.  This 
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model  was  subsequently  expanded  by  the  team  to  develop  the  value  systems  hierarchy 
(see  Chapter  II  section  C,  Value  Systems  Design).  The  hierarchy  of  JC3M  was 
constructed  based  on  three  major  functions  in  the  evaluation  of  a  C4I  SoS.  The  first 
function,  Planning  the  C4I  SoS  Evaluation,  was  the  focus  of  JC3M.  The  second  function, 
Conducting  the  C4I  SoS  Evaluation,  as  well  as  the  third  function,  Reporting  the  Results 
of  the  C4I  SoS  Evaluation,  are  not  modeled,  for  reasons  stated  above. 

From  the  Plan  function,  the  JC3M  team  decomposed  sub-functions  (Define 
Problem,  Identify  Systems,  Define  Criteria  and  Plan  Review).  This  analysis  identified 
planned  inputs  and  outputs  from  each  function  and  sub-function.  Figure  8  illustrates  the 
major  functions  of  JC3M. 


Figure  8.  JC3M  Functions. 

Development  of  JC3M  focused  on  Planning  a  C4I  SoS  Evaluation.  This  figure  illustrates 
the  decomposition  of  subfunctions  within  the  JC3M  Planning  Process.  Conducting  a  C4I 
SoS  evaluation  (2.0)  and  Reporting  the  Results  of  the  Evaluation  (3.0)  were  recognized 
as  well-functioning  processes  at  C4I  test  organizations;  JC3M  proposed  utilization  of 
these  existing  processes,  rather  than  developing  new  processes. 

3.  JC3M  Inputs  and  Outputs 

Inputs  to  the  JC3M  system  include  stakeholder  requirements,  C4I  SoS 
components,  and  labor.  Outputs  of  the  JC3M  system  include  the  assessment  plan  for  the 
C4I  SoS  and  capabilities  under  review;  EM;  and  a  SE  Artifact  report,  which  describes  the 
current  state  of  requirements  documentation  for  the  SoS  and  components  under  review. 
One  of  the  secondary  outputs  includes  a  documented  C4I  SoS  configuration  with 
documented  capabilities,  which  in  turn  can  provide  war-fighters  with  a  known  baseline 
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configuration  for  implementation.  If  utilized,  this  baseline  increases  war-fighter 
confidence  in  the  effectiveness  of  the  C4I  SoS,  and  reduces  the  resources  required  for 
C4I  SoS  implementation,  use  of  the  SoS,  training,  troubleshooting,  and  life  cycle  support. 
While  the  assessment  plan  is  a  by-product,  the  directive  portion  of  the  assessment  plan 
(“in  order  to  assess  the  capability  to  perform  Task  X. . .  the  System  Y  operator  will 
initiate  a  request  for  support  by  sending  message  1  via  function  2  from  screen  3. . .  ”)  can 
be  used  as  a  “how-to”  list  for  SoS  operators. 

C.  VALUE  SYSTEM  DESIGN 

Value  System  Design  was  the  process  used  to  create  a  qualitative  and  quantitative 
model  [Paulo  (c),  2005]  which  portrayed  the  system  functions,  objectives,  and  evaluation 
measures.  The  quantitative  model  was  used  to  analyze  and  evaluate  the  performance  of 
the  alternative  solutions.  Value  System  Design  was  conducted  by  Value  System 
Modeling  and  creating  a  Value  Hierarchy. 

The  project  team  found  the  description  “. . .  developing  a  good  set  of  MOEs 
[Measures  of  Effectiveness]  is  usually  a  harrowing  business”  [Feuchter,  2000:  40]  an 
accurate  account  of  the  process. 

1.  Value  System  Modeling 

The  value  system  of  JC3M  system  was  constructed  based  on  a  three-phase  model 
of  C4I  SoS  performance  evaluation:  Plan  the  execution  of  the  evaluation,  Conduct  the 
evaluation,  and  Report  on  the  evaluation.  This  model  was  constructed  based  on  JC3M 
team  experience  with  C4I  SoS  evaluation. 

From  the  top-level  functions  (Plan,  Conduct,  Report)  the  JC3M  team  identified 
sub-functions,  and  an  objective  was  identified  for  each  of  the  sub-functions.  After 
objectives  were  established,  each  was  reviewed  to  determine  an  appropriate  level  of 
performance  which  was  used  in  comparing  the  alternative  solutions  [Feuchter,  2000:  39- 
40],  The  overall  goal  was  to  be  able  to  drill  down  from  effective  need  for  JC3M,  to  get  to 
the  “what”  that  JC3M  was  designed  to  do,  and  eventually  to  the  “how  well”  standards  of 
performance.  As  seen  in  Figure  9,  Feuchter  calls  the  “how  well”  factors  MOE;  the  JC3M 
team  refers  to  these  as  EM. 
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Identify  Goals 


Identify  Tasks  to  be  Performed 
to  Meet  Goals 


Identify  MOEs  that  Measure 
How  Well  Each  Alternative 
Performs  Each  Task 


Figure  9.  Developing  Evaluation  Measures 

Evaluation  Measures  Trace  Back  to  System  Goals.  The  complete  Value  System  Model 
process  and  results  are  included  in  Appendix  E.  Most  EM  have  a  single  quantifiable 
factor,  but  some  have  multiple  factors.  A  graphical  layout  of  the  Value  System  Hierarchy 
is  also  included  as  Appendix  E  [From  Feuchter,  2000:40]. 

2.  Value  Hierarchy 

The  JC3M  team  determined  functional  analysis  [Blanchard  and  Fabrycky,  1998: 
26]  was  required  to  ensure  stakeholder  requirements  were  incorporated  in  the  final  JC3M 
system.  Stakeholder  requirements  were  collected  (Appendix  C  -  Questionnaires),  and  a 
value  hierarchy  [Paulo  (b),  2005]  was  created  to  translate  stakeholder  requirements  into 
design  criteria  [Blanchard  and  Fabrycky,  1998:  29]  for  the  JC3M  system.  The  initial 
decomposition  of  system  requirements  is  displayed  in  Figure  10. 


Figure  10.  JC3M  Top  Level  Value  Hierarchy. 

Top  level  value  hierarchy  for  the  JC3M  system  after  stakeholder  input.  As  described, 
JC3M  will  focus  on  the  functional  requirements  in  section  1 .0,  and  utilize  existing 
processes  and  methodologies  for  2.0  and  3.0. 


24 


The  team  developed  the  value  hierarchy  with  both  functional  (1.0,  2.0,  3.0) 
requirements  and  non-functional  (4.0,  5.0,  and  6.0)  requirements  [Buede,  2000:  39]. 
From  this  initial  level  of  requirements,  more  discrete  sub-functions  were  identified  by 
reviewing  stakeholder  expressed  and  implied  requirements.  The  complete  JC3M  value 
hierarchy  and  all  measures  are  provided  at  Appendix  E.  Figure  1 1  shows  the  critical 
measures  in  white  boxes. 


Figure  11.  JC3M  Functional  Hierarchy. 

This  figure  illustrates  selected  JC3M  subfunctions  based  on  the  JC3M  Value  Hierarchy 
(illustrated  Figure  10).  Figure  1 1  also  illustrates  the  traceability  of  EM  to  JC3M 
functional  and  non-functional  requirements. 

3.  Critical  Evaluation  Measures 

EM  were  developed  for  each  function  of  the  JC3M  system,  to  ensure  each 
function  was  not  only  measurable,  but  potentially  capable  of  serving  as  a  discriminator  in 


25 


considering  alternative  JC3M  solutions.  This  section  provides  a  summary  of  the  critical 
EM  to  be  used  in  analyzing  JC3M  alternative  solutions.  A  detailed  description  of  all 
JC3M  EM  is  provided  in  Appendix  E 

Table  1  identifies  the  critical  EM  used  to  compare  the  performance  of  each 
alternative;  cost  was  calculated  separately,  and  later  compared  to  the  overall  performance 
of  each  alternative.  The  body  of  the  table  contains  the  definition  of  each  evaluation 
measure,  which  describes  how  the  EM  was  calculated.  The  range  and  format  of  expected 
values  of  the  EM  are  defined,  and  the  rationale  and  relevance  describes  what  the  EM 


measured,  and  why  the  EM  was  used  to  compare  alternatives. 


Percentage  of 
Traceable 
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Plan 

Evaluation 

Quality  of 

Planning 

Products 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

JC3M 

Function 

Define  Measures 
1.3.2 

Planning 

Results 

1.4.3 

Planning 

Results 

1.4.3 

Input  System 
Flexibility 

4.1 

Input  System 
Flexibility 

4.1 

Definition 

Alternative 

generated 

measures,  traceable 
to  stakeholder 
requirements, 
divided  by  the 
number  of 
measures  generated 
by  the  alternative. 

Elapsed  time 
(in  days)  of 
planning  for 

C4I  SoS 
evaluation 

Assign  an 
overall  quality 
level  to  the 
planning 
documents 
produced. 

Divide  percent 
change  in  labor 
hours  to  conduct 
planning  phase 
of  JC3M  by  the 
percent  change 
in  systems  under 
test. 

Divide  percent 
change  in  duration 
to  conduct 
planning  phase  of 
JC3M  by  the 
percent  change  in 
systems  under 
test. 

Ratio  level  data, 
from  0  -  100% 

Ratio  level  data 
>  0  days 

Ratio  level  data 
from  1-4 

Ratio  level  data 
from  0  -  oo 

Ratio  level  data 
from  0  -  oo 

Rationale 

and 

Relevance 

Identifies 
objectivity  of 
performance 
measures. 

Predicts  SoS 
evaluations  that 
can  be 

conducted  in  a 

Identifies 
predicted 
utility  of 
alternative. 

Predicts  changes 
in  cost  of  SoS 
evaluation  based 
on  size. 

Predicts  changes 
in  duration  of  SoS 
evaluation  based 
on  size. 

Performance 
measures  traceable 
to  doctrine  will  be 
perceived  as 
objective, 
increasing  the 
value  of  the 
evaluation. 

year. 

Alternatives 
that  permit 
multiple  SoS 
evaluations 
generate  data  to 
support  fielding 
decisions 

sooner. 

Quality  of  the 
planning 
products  drives 
the  overall 
value  of  the 
alternative. 

Can  be  used  to 
determine  most 
effective 
alternative  based 
on  SoS  size. 

Can  be  used  to 
determine  most 
effective 
alternative  based 
on  SoS  size. 

Table  1.  JC3M  Critical  Evaluation  Measures. 

This  table  illustrates  Critical  EM  used  to  compare  the  performance  of  alternative 
solutions.  Dimension  of  the  measure  (Nominal,  Ordinal,  Interval,  or  Ratio-level  data)  is 
provided.  Rationale  for,  and  relevance  of,  the  EM  is  also  provided.  Note  “Quality  of 
Planning  Products”  is  expressed  on  a  Likert  Scale  as  described  in  Appendix  C. 
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III.  DESIGN  ALTERNATIVES 


The  goal  of  the  Design  Alternatives  phase  was  to  provide  possible  JC3M 
solutions  that  were  broad  enough  in  approach  to  present  stakeholders  with  a  viable  range 
of  approaches  to  the  problem,  yet  few  enough  in  number  so  as  to  be  capable  of  analysis 
within  project  constraints.  The  Design  Alternatives  phase  consisted  of  three  tasks: 
Alternative  Generation,  Feasibility  Screening,  and  establishing  Criteria  for  Scoring. 

Alternative  Generation  took  the  outputs  from  the  Problem  Refinement  phase, 
including  the  problem  refinement  report  and  JC3M  value  hierarchy  with  evaluation 
measures,  and  created  a  range  of  alternative  solutions  to  focus  on  solving  the  JC3M 
problem  statement  -  identifying  objective  threshold  values  for  measuring  SoS 
performance.  Alternative  Generation  also  included  the  consideration  of  known 
alternatives:  the  status  quo,  a  near-term  future  solution,  and  a  long-term  future  solution. 
With  three  known  alternatives,  the  team  generated  two  new  alternatives  to  present  a 
sufficiently  broad  range,  yet  still  maintain  the  ability  to  conduct  a  thorough  analysis. 

Feasibility  Screening  was  used  to  evaluate  the  variety  of  alternatives,  and  identify 
those  which  appeared  to  be  feasible,  different  from  the  baseline  and  other  alternatives, 
and  few  enough  in  number  to  be  effectively  analyzed  within  the  scope  of  the  project.  The 
output  of  this  process  was  a  smaller  set  of  feasible  alternatives. 

Criteria  for  Scoring  were  established  based  on  EM,  problem  refinement  report, 
and  the  JC3M  value  hierarchy  tree  in  order  to  identify  those  criteria  which  would  be  later 
used  for  comparing  the  alternatives.  This  process  also  identified  those  EM  which  would 
be  drawn  from  the  simulations  of  the  alternatives.  Outputs  of  this  process  included  the 
scoring  criteria  identified,  as  well  as  a  blank  quantitative  modeling  decision  matrix. 

A.  ALTERNATIVE  GENERATION 

The  team  identified  known  alternatives  for  inclusion  in  the  set  of  possible  JC3M 
solutions  by  reviewing  historical  processes,  interviewing  current  stakeholders,  and 
participating  in  new  system  development. 
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1.  Known  Alternatives 

a.  Federation  of  Systems 

The  first  known  alternative  was  the  FEDeration  Of  Systems  (FEDOS) 
process  used  at  MCTSSA.  FEDOS  was  a  user-driven  process,  with  C4I  program 
managers  collectively  defining  the  configuration  of  the  C4I  SoS  [Manning  (c),  2007], 

FEDOS  was  designed  to  assess  the  performance  of  C4I  systems  when 
assembled  into  the  Marine  Air  Ground  Task  Force  (MAGTF)  C4I  SoS.  FEDOS  began  at 
the  order  of  the  Deputy  Commander  for  C4I  Integration  and  Interoperability  (C4II)  at 
MARCORSYSCOM,  who  tasked  MCTSSA  to  assess  MAGTF  C4I  SoS  and  systems 
interoperability.  FEDOS  was  created  because  there  were  processes  to  assess  the 
interoperability  and  performance  of  a  C4I  system,  but  none  to  assess  the  performance  or 
interoperability  of  the  C4I  SoS.  The  Deputy  Commander  for  C4II  assembled  a  working 
group  of  stakeholders  from  the  C4I  system  community  at  MARCORSYSCOM,  and 
tasked  MCTSSA  to  assign  a  test  director.  The  working  group  decided  what  systems 
would  participate,  what  requirements  were  to  be  tested,  and  the  schedule  of  events  to 
include  test  planning,  test  conduct,  and  results  reporting. 

Because  FEDOS  provided  an  opportunity  to  evaluate  the  performance  of 
C4I  systems  in  the  SoS,  it  was  also  used  to  evaluate  performance  of  developmental 
systems.  CONDOR,  as  an  example,  was  developed  to  provide  communications 
connectivity  to  units  physically  distant  from  stationary  long  haul  communications  paths. 
CONDOR  was  included  in  a  June  2005  FEDOS  event,  prior  to  deployment  to  Iraq  in 
September  2005. 

The  MARCORSYSCOM  Product  Groups,  responsible  for  developing, 
fielding,  and  supporting  C4I  systems,  were  not  required  by  order  or  doctrine  to 
participate  in  FEDOS,  and  a  passing  grade  from  FEDOS  was  not  a  requirement  for  a 
milestone  decision.  This  led  to  the  perception  of  FEDOS  as  a  cost,  schedule,  and 
performance  risk,  which  in  turn  led  to  ineffective  participation  from  acquisition 
community  representatives. 
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Because  the  MAGTF  C4I  SoS  was  not  designed  in  compliance  with 
architecture,  there  are  no  overarching  C4I  SoS  performance  measures  or  threshold 
criteria.  This  lack  of  doctrinal  C4I  SoS  performance  criteria  meant  that  MCTSSA  test 
personnel  had  to  engage  in  long  and  at  times  inconclusive  negotiations  with  stakeholders 
in  order  to  define  the  threshold  values  that  were  used  to  measure  SoS  performance,  and 
determine  if  components  passed  or  failed  the  test.  The  Product  Groups  viewed  the 
results  of  FEDOS  askance,  as  the  event  results  were  not  tied  to  objective  SoS-level 
performance  requirements. 

Further,  FEDOS  was  perceived  as  a  no-win  situation  for  Product  Groups. 
After  a  system  had  successfully  passed  Development  and  Operational  Tests,  and 
demonstrated  compliance  with  system-level  performance  requirements  as  described  in 
the  Capability  Development  Document  (CDD),  Operational  Requirements  Document 
(ORD),  or  equivalent,  FEDOS  tested  component  systems  in  ways  they  had  not  been 
designed  to  be  used. 

The  conduct  of  FEDOS,  outlined  in  Figure  12,  was  organized  in  a  series  of 
events,  each  of  which  culminated  in  a  decision  point  known  as  a  “Control  Gate.”  Each 
gate  represented  a  milestone  that  had  to  be  reached  before  the  planning  effort  could 
continue.  The  gates  included  agreements  on  what  was  under  test,  how  the  network  would 
be  configured,  how  the  systems  would  be  evaluated,  what  specific  test  procedures  would 
be  used,  interpretation  of  results,  and  the  contents  of  the  test  plan. 

Both  execution  and  reporting  of  SoS  evaluation  results  have  effective 
processes  in  place;  JC3M  focuses  on  the  planning  of  SoS  evaluations.  For  the  purposes 
of  this  project,  only  the  initial,  Planning  phase  of  FEDOS  was  considered  as  a  candidate 
alternative  JC3M  solution.  This  phase  consisted  of  three  tasks  as  discussed  below. 

In  Task  1,  the  team  elicited  test  requirements.  This  was  done  by  analysis 
of  stakeholder  inputs  and  identified  hardware  and  software  versions,  test  schedule 
requirements,  and  functional  test  requirements.  The  functional  requirements  were 
determined  based  on  the  SoS  capabilities  that  were  to  be  exercised  in  order  to  evaluate 
the  SoS  as  requested.  Additionally,  documentation  and  data  collection  requirements  were 
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identified,  as  well  as  customer  communications  requirements.  After  stakeholder  review, 
changes  were  incorporated  into  a  final  list  of  test  requirements  to  be  presented  at  the 
control  gate  exit  from  Task  1.  This  requirements  list  included  a  draft  SoS  components 
list,  draft  project  schedule,  draft  functional  requirements  and  architecture,  draft 
documentation  and  data  collection  plan,  and  draft  customer  communications  plan.  Figure 
12  provides  an  illustration  of  the  serial  processes  in  FEDOS.  Figure  13  provides  a 
detailed  view  of  FEDOS  Task  1. 

Task  2  generated  an  evaluation  plan.  The  FEDOS  team  reviewed  Task  1 
outputs  and  generated  draft  Test  Thread  Architecture  diagrams,  and  reviewed  with 
stakeholders.  As  stakeholder  approval  was  received  for  the  test  thread  architecture,  draft 
test  thread  descriptions  were  generated.  Draft  metrics  and  a  grading  process  were 
generated  and  reviewed  with  stakeholders.  Finally,  a  draft  Evaluation  Plan,  containing 
the  draft  architecture  diagrams,  test  thread  descriptions,  and  grading  process  was 
assembled  and  reviewed  with  stakeholders.  After  review,  incorporation  of  requested 
changes,  and  assembly  of  the  final  evaluation  plan,  the  stakeholders  had  to  agree  to  the 
evaluation  plan  as  another  control  gate. 

Task  3  generated  a  test  plan  and  test  procedures.  The  team  created  a  draft 
test  plan,  incorporating  outputs  from  Task  1  (requirements  list)  and  Task  2  (evaluation 
plan).  A  test  procedure  format  was  chosen,  reviewed  with  stakeholders,  and  used  to 
document  the  test  procedures  proposed  for  the  event.  Both  the  test  plan  and  procedures 
were  reviewed  in  detail  and  presented  to  stakeholders  for  approval.  Once  the  test  plan 
and  the  test  procedures  were  approved,  the  planning  phase  of  FEDOS  was  concluded. 

Because  FEDOS  was  the  only  alternative  solution  that  had  been  used  by  a 
C4I  test  organization,  it  was  considered  the  “status  quo”  or  baseline  JC3M  alternative 
solution. 
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Output 


7  High  Level  Tasks  to 
complete  a  test  event 


Control  Gate 

# 


Elicit  Test 
Requirements 
TASK  #1 


Hardware/Software  Version  Requirements, 
Test  Schedule  Requirements,  Functional  Test 
Requirements,  Deliverable  Requirements, 
Customer  Communication  Requirements 


Planning 


Generate  an 
Evaluation  Plan 
TASK  #2 


Evaluation  Plan,  SSR 


Develop  a  Test  Plan 
and  Test  Procedures 
v  TASK  #3 


Environmental 

Preparation 


TASK  #4 


Redlined  Architecture  Diagrams, 
Configuration  Cut  Sheets 


Execution 


Execute  a  Dry  Run 
TASK  #5 


Dry  Run  Report,  Daily  Debriefs,  Redlined  Test 
Procedures 


Execute  a  Formal 
Evaluation 
TASK  #6 


Reporting 


Analyze  Results  and 
create  a  Test  Report 
TASK  #7 


Test  Report,  Lessons  Learned 


Figure  12.  FEDOS  Process. 

FEDOS  has  7  tasks  to  complete  a  test  event;  JC3M  will  consider  the  Planning  Phase 
(Tasks  1  through  3)  of  FEDOS  as  an  alternative  JC3M  solution.  Each  task  (identified  in 
a  block)  has  outputs  (listed  in  an  oval).  Each  task  proceeds  when  a  control  gate 
(identified  by  a  numbered  circle)  is  completed.  (See  Figure  13  for  a  detailed  example  of 
FEDOS  Task  1.) 
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FEDOS  phases  were  executed  in  a  serial  manner,  requiring  explicit 
approval  from  all  stakeholders  before  commencement  of  subsequent  tasks.  The  serial 
nature  of  FEDOS  can  be  seen  in  Figure  13,  which  illustrates  only  Task  1  of  the  FEDOS 
process. 


Figure  13.  FEDOS  Task  1  Elicit  Test  Requirements. 

This  IDEFO  illustration  of  the  FEDOS  “Elicit  Test  Requirements”  task  portrays  the 
subtasks  (“Elicit  Flardware  and  Software  Version  Requirements”)  used  to  generate  the 
outputs  identified  in  Figure  12.  Each  process  contained  exit  criteria  used  as  a  control 
gate;  all  stakeholders  had  to  agree  on  the  outputs  of  a  process  before  the  next  process 
started.  IDEFO  illustrations  of  each  alternative  are  provided  at  Appendix  F. 

b.  Marine  Air  Ground  Task  Force  Command,  Control, 
Communications,  Computers,  and  Intelligence  Capability 
Certification  Test 

MC3T,  a  system  for  defining,  documenting,  and  evaluating  the 
performance  of  a  MAGTF  C4I  SoS  [Finn,  2007],  was  identified  as  the  second  known 
alternative.  MC3T  was  implemented  as  a  replacement  for  FEDOS  at  MCTSSA. 

MCTSSA  was  charged  by  MARCORSYSCOM  with  developing, 
planning,  executing,  and  reporting  on  a  proof  of  concept  event  to  be  conducted  in 
October  2007.  Three  members  of  this  Capstone  project  team  have  been  actively  engaged 

in  the  development  and  implementation  of  MC3T,  and  will  continue  to  support  MC3T  as 
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planning,  execution,  and  reporting  are  conducted.  Other  MC3T  participants  include  U.S. 
Navy  Space  and  Naval  Warfare  Center  (SPAWAR)  Systems  Center,  Charleston,  S.C.  and 
U.S.  Marine  Corps  Combat  Development  Command  (MCCDC). 

The  first  MC3T  event  will  not  develop  C4I  SoS  measures  of  performance 
for  consideration,  due  to  time  constraints,  and  will  instead  be  used  to  prove  the  validity  of 
MC3T  processes.  During  and  after  the  first  iteration  of  MC3T,  stakeholders  will  review 
results  to  determine  if  this  system  will  be  continued,  modified,  or  replaced.  Because  this 
review  period  will  occur  shortly  after  the  delivery  of  this  JC3M  final  report,  MC3T 
stakeholders  are  interested  in  reviewing  JC3M  results  for  possible  inclusion  in  future 
MC3T  events. 

MC3T  has  been  developed  by  an  Integrated  Product  Team  (IPT)  made  up 
of  stakeholders  from  MCTSSA  as  well  as  representatives  of  the  MARCORSYSCOM 
Product  Groups.  Product  Group  representatives  defined  a  "Capabilities  Package" 
complete  with  System  requirements,  and  DoD  Architecture  Framework  Operational 
views  and  System  views  that  depicted  the  systems  under  their  cognizance.  MCTSSA 
analyzed  the  Capabilities  Package  and  produced  a  Consolidated  Requirements 
Assessment  (CRA).  The  CRA  was  an  agreement  between  the  stakeholders  on  what 
needed  to  be  tested,  what  the  required  resources  were,  what  the  Information  Assurance 
compliance  requirements  were,  and  what  the  draft  test  dates  were.  Once  the  CRA  was 
approved,  MCTSSA  produced  a  Technical  Proposal. 

The  Technical  Proposal  defined  the  technical  solution  the  IPT  proposed  to 
meet  the  requirements  in  the  CRA  in:  staffing,  C4I  systems  architecture  design, 
monitoring  network  architecture  design,  test  cases,  data  capture  and  analysis  plan, 
information  assurance  plan,  and  risk  assessment. 

Once  the  Technical  Proposal  was  confirmed,  it  became  the  Technical 
Solution.  The  Technical  Solution  made  up  approximately  90%  of  the  Test  Plan  and 
included  detailed  test  procedures  and  additional  reference  documentation. 
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c.  Joint  Test  and  Evaluation  Methodology  Capability  Test 

Methodology 

The  JTEM  CTM  was  chosen  as  the  third  alternative.  The  purpose  of 

JTEM  is  to 

...develop,  test,  and  evaluate  M&P  [Methods  and  Processes]  for  defining 
and  using  a  distributed  LVC  [Live,  Virtual,  and  Constructive]  joint  test 
environment  to  evaluate  system  performance  and  joint  mission 
effectiveness.  JTEM  will  focus  on  developing  and  enhancing  M&P  for 
designing  and  executing  tests  of  SoS...  [JTEM  Rock  Drill  Event  Final 
Report,  2007:  i]. 

The  five  phases  of  the  JTEM  CTM  are  Characterize  Test;  Plan  Test; 
Implement  Live,  Virtual,  Constructive  Distributed  Environment;  and  Execute  Test 
[JTEM  CTM  M&P  Model  Description,  2007:  4-5],  Phases  1  and  2  (Characterize  Test 
and  Plan  Test)  are  considered  as  a  combined  alternative  JC3M  solution  and  are  described 
below. 


CTM.l  Characterize  Test 

“Initial  planning  negotiations  concerning  test  program  concepts  and  test 
capabilities  are  conducted  during  the  Characterize  Test  phase.  Program  characterization 
processes  include  developing  the  joint  operational  context  for  testing,  developing  the  test 
concept,  and  developing  the  test’s  evaluation  strategy.  Test  capability  characterization 
processes  include  technical  assessment  producing  an  initial  LVC  distributed  environment 
test  design  and  programmatic  assessment  involving  high-level  test  scheduling  and  test 
resource  estimates.  Information  product  inputs  to  Characterize  Test  relevant  to 
developing  joint  operational  context  for  testing  include  the  critical  CDD,  as  well  as  the 
Initial  Capabilities  Document  (ICD),  JOpsC  Family  products,  the  Universal  Joint  Task 
List  (UJTL),  Defense  Planning  Scenarios  (DPS),  Combatant  Command  (COCOM) 
operation  plans/orders  (OPLAN/OPORD),  and  the  Test  and  Evaluation  Master  Plan 
(TEMP).  Characterize  Test  product  inputs  relevant  to  test  concept  and  evaluation 
development  include  an  approved  AoA  plan,  test  and  evaluation  strategy,  TEMP,  and 
operational  test  plan.  The  Characterize  Test  phase  produces  a  test  program  introduction 
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document  (PID)  and  statement  of  capabilities  (SOC)”  [ITEM  CTM  M&P  Model 
Description,  2007:  4], 

CTM.2  Plan  Test 

“During  the  Plan  Test  phase,  test  concepts  ...  are  further  developed  into  a 
test  plan.  Test  Planning  processes  include  developing  the  test  design,  performing  . . . 
environment  analysis,  and  coordinating  test  support,  and  the  synthesis  of  these  processes 
by  developing  the  overall  test  plan.  Developing  the  test  design  involves  producing  test 
vignettes  and  a  data  management  and  analysis  plan  (DMAP).  Performing  . . . 
environment  analysis  produces  a  ...  Functional  Description.  Also,  test  support 
coordination  produces  test  support  plan  products.  The  Plan  Test  process  then  produces 
an  overall  Test  Plan,  incorporating  all  identified  products  from  this  phase. 

“This  Level  I  division  into  separate  process  phases  is  somewhat  of  an 
abstraction  and  a  simplification  of  what  occurs  in  a  real  test  cycle.  Many  of  the  Level  I 
phases  and  lower  level  processes  are  iterative  in  nature.  Many  processes  are  performed 
in  parallel,  and  the  results  of  these  processes  are  fed  back  into  the  iterative  work  of  other 
processes”  [JTEM  CTM  M&P  Model  Description,  2007:  4], 

The  JTEM  CTM  was  tested  in  a  limited  scope  during  2007,  but  the 
development  of  the  CTM  will  not  be  complete  until  2009.  The  CTM  will  not  be  executed 
during  the  conduct  of  the  JC3M  project.  The  team  considered  phases  1  and  2  as  an 
alternative  JC3M  solution.  All  five  phases  of  the  JTEM  CTM  are  illustrated  in  Figure  14. 
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Figure  14.  JTEM  CTM  Phases 

JTEM  CTM  processes  are  pictured;  JC3M  will  consider  the  first  two  phases 
(Characterize  and  Plan)  as  an  alternative  solution.  Dark  Blue  boxes  are  CTM  phases; 

Green,  round  edge  boxes  are  inputs  and  outputs;  Execute  Test  is  identified  in  yellow  as  a 
simulated  event  performed  in  JTEM  CTM  development.  CTM  1  and  CTM  2  are 
considered  as  a  JC3M  alternative  solution  [From  JTEM  CTM  M&P  Model  Description, 

2007:  4], 

2.  Generating  New  Alternatives 

Systems  Engineering  practice  and  critical  thinking  [Paul  et  al.,  2006:  23]  call  for 
exploration  of  a  full  range  of  options  in  order  to  provide  good  solutions.  Alternative 
solutions  must  be  evaluated  through  a  fair  minded  [Paul  et  al.,  2006:  6]  and  logical 


36 


process,  using  quantitative  and  qualitative  methods  to  determine  the  optimum  solution  for 
the  JC3M  system. 

The  known  alternatives  discussed  above  represent  the  current  state  of  C4I  SoS 
evaluation  systems,  and  reflect  the  resource,  risk,  and  organizational  constraints  in  effect 
at  the  creation  of  these  systems.  The  JC3M  team  generated  new  alternatives  to  provide  a 
fresh  perspective  on  the  problem,  in  an  attempt  to  generate  a  better  range  of  solutions. 

The  team  determined  that  two  new  alternatives,  in  addition  to  the  three  known 
alternatives,  would  provide  stakeholders  a  meaningful  range  of  potential  solutions  to 
consider.  Limiting  the  alternatives  to  five  would  also  limit  the  modeling,  simulation,  cost 
estimation,  and  analysis  tasks  to  a  scope  that  could  be  managed  over  the  duration  of  the 
project. 

To  produce  the  new  alternatives  the  team  utilized  a  morphological  box  [Zwicky 
and  Wilson,  1967:  9]  as  a  tool  to  both  guide  the  process  and  record  the  results. 

3.  Morphological  Box  Process 

The  morphological  box  (“Zwicky  Box”)  process  began  by  defining  the  parameters 
of  importance  to  the  problem.  For  the  problem  under  consideration,  the  JC3M  functional 
hierarchy  was  used  to  define  the  critical  parameters  to  be  investigated.  These  top  level 
functions  were  Define  the  Problem,  ID  Systems  Under  Test  (Components  of  the  SoS), 
Define  Criteria,  and  Ensure  Readiness. 

A  matrix,  containing  all  the  potential  solutions  of  the  problem,  was  created.  The 
team  used  brainstorming  to  generate  potential  solutions  for  each  separate  JC3M  system 
function.  The  brainstorming  process  consisted  of  proposing  potential  solutions  for  each 
separate  JC3M  function  without  consideration  of  cost,  for  feasibility,  accuracy,  speed,  or 
any  other  factor.  Ideas  were  not  evaluated,  but  recorded  in  a  non-graded  manner  under 
the  respective  functional  column  of  the  alternative  matrix.  This  process  was  repeated  for 
each  JC3M  system  function  until  the  team  could  not  identify  or  generate  any  additional 
concepts. 
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All  of  the  generated  potential  functional  solutions  were  evaluated,  and  after 
clearly  redundant  entries  were  eliminated,  the  remaining  potential  function  solutions  were 


as  depicted  in  Table  2. 


Define  the  Problem 

ID  Systems  Under  Test 

Define  Criteria 

Ensure 

Evaluation 

Readiness 

Have  Subject  Matter 
Experts  do  it 

Engineering  Document  review 

Experience  from 
earlier  test 

Test  Manager 
Review 

Get  from  CDD 

Depart  of  Defence  Architecture 
Framework  (DoDAF) 
Document  Review 

Ask  JITC 

Program  Manager 
(PM)  Review 

Conduct  survey 

Previous  Systems  Under  Test 

Ask  users 

Cost  Driven 

Test  agency  defines 

Ask  JITC 

"Trouble  Calls" 
recorded  on  issues 

All  Stakeholders 
Review 

JITC  defines 

ORD/CDD  review 

Build  from  SoS 
Capabilities 

Planning  is  finished 

Stakeholders  define 

Field  Concept  of  Operations 

Look  at  Peers  (Joint, 
Service) 

Schedule  Driven 

Acquisition  manager 
defines 

What  PM  requested 

What  PM  asked  for 

Subject  Matter 
Expert  Review 

JTEM  defines 

Changes  to  SoS  from  last  test 

Changed  capability 
or  function  in  SoS 

Dry  Run  Test  (and 
see  if  it  works) 

Ad  hoc  definition 

Functional  Breakdown 

Work  Breakdown 
Structure 

Modeling  of  test 
plan 

Last  definition  used 
in  previous  test 

Bottom  up  review 

Hardware  or 
Software  Upgrade 

Run-Test-Run 

Users  define 

Test  everything 

Ask  SPAWAR 

JITC  Definition 

User  requirements 

Whatever  changed 

Test  everything 

SAR  Review 

What  is  changed  in 
SoS 

Test  agency  determines 

"Just  Do  It"  (no 
specific  guidance) 

ORD/CDD 
Crosswalk  , 

Current  issues  seen 
by  SoS  users 

System  Anomaly  Report  review 
(SAR) 

ORD/CDD  Review 

Change  in  Test  Plan 
from  previous  event 

(SAR)  Generation 

SAR  System  ID 

Trouble  calls  from 
the  field 

Don't  Define  It 

Stakeholder  Review 

Table  2.  JC3M  Morphological  Box. 

Potential  solutions  identified  for  each  JC3M  sub-function,  as  generated  by  Zwicky  Box 
process.  Top  Row  is  the  list  of  functions,  and  each  cell  below  it  represents  a  proposed 
solution  to  that  issue,  in  isolation. 
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Once  the  team  had  identified  a  number  of  potential  solutions  for  each  function, 
the  separate  potential  solutions  had  to  be  assembled  into  a  cohesive  package  for 
consideration  as  an  alternative.  One  goal  of  this  stage  was  to  identify  alternatives  that  are 
distinctly  different.  Another  goal  of  this  stage  was  to  identify  those  approaches  which  are 
attractive,  viable,  or  possible,  and  to  eliminate  the  approaches  which  are  not  attractive, 
viable,  or  possible. 

The  team  considered  each  potential  solution  to  a  function  separately,  and  then 
compared  the  list  of  potential  solutions  for  the  next  function  in  series,  looking  for 
similarities  in  theme.  As  an  example,  for  the  “Define  Problem”  function,  one  proposed 
approach  was  to  consider  “What  is  changed  in  SoS.”  In  order  to  perform  the  “ID 
Systems  under  test”  function,  one  proposed  approach  was  to  consider  “Whatever 
changed.”  For  the  “Define  Criteria”  function,  one  proposed  approach  was  to  consider 
“Changed  capability  or  function  in  the  SoS.”  For  the  “Ensure  Evaluation  Readiness” 
function,  one  approach  was  to  consider  “Change  in  Test  Plan  from  previous  event.”  All 
these  approaches  had  the  common  theme  of  reviewing  only  what  had  changed  from  a 
previous  evaluation,  so  these  were  assembled  into  an  alternative.  The  name  for  this 
alternative  was  assigned  after  members  of  the  team  proposed  that  the  underlying  theory 
was  to  consider  “what  has  changed.” 

After  review  of  all  potential  solutions,  nine  different  alternatives  were  created  by 
linking  themes.  Table  3  below  shows  the  nine  alternatives,  named  for  the  theme 
underlying  the  individual  approaches.  A  detailed  description  of  the  proposed  alternatives 
is  included  in  Appendix  D. 


39 


Alternative 

Name 

Define  Problem 

ID  Systems  Under 
Test 

Define  Criteria 

Ensure 

Evaluation 

Readiness 

Exhaustive 

Don’t  Define  It 

Test  everything 

’’Just  Do  It”  (no 

specific 

guidance) 

Run-Test-Run 

User  Defines 

Stakeholders-defme 

What  PM  requested 

Ask  users 

All 

Stakeholders 

Review 

Do  No  Harm 

JITC  defines 

Ask  JITC 

Ask  JITC 

JITC  Definition 

System 

Anomaly 

Report 

SAR  Generation 

SAR  System  ID 

Trouble  calls 
from  the  field 

SAR  Review 

Deliberate 

Method 

Stakeholders  define 

Functional  Breakdown 

Stakeholder 

Review 

Modeling  of 
test  plan 

Capabilities 

Documentation 

Get  from  CDD 

ORD/CDD  review 

ORD/CDD 

Review 

ORD/CDD 

Crosswalk 

Program 

Manager 

Direction 

Acquisition  manager 
defines 

What  PM  requested 

What  PM  asked 
for 

PM  Review 

Test  Agency 
Direction 

Test  agency  defines 

Test  agency 
determines 

Experience  from 
earlier  test 

Test  Manager 
Review 

Change  Driven 

What  is  changed  in 

SoS 

Whatever  changed 

Changed 
capability  or 
function  in  SoS 

Change  in  Test 
Plan  from 
previous  event 

Table  3.  JC3M  Alternatives. 

Initial  JC3M  potential  solutions,  based  on  JC3M  sub- functions,  and  generated  by  Zwicky 
box  process.  Proposed  alternatives  are  composed  of  like  approaches  to  sub-function 
challenges. 

B.  FEASIBILITY  SCREENING 

After  alternative  generation,  the  JC3M  project  team  reviewed  the  group  of 
potential  alternatives  to  eliminate  those  which  were  clearly  infeasible,  in  order  to  reduce 
wasted  effort.  The  team  also  reviewed  the  group  of  potential  alternatives  to  eliminate 
those  which  were  very  similar,  in  order  to  present  stakeholders  with  feasible  alternatives 
that  were  sufficiently  different  so  as  to  provide  a  variety  of  approaches. 

After  reviewing  the  nine  alternatives  from  the  standpoint  of  feasibility  and 
similarity,  the  User  Defines  alternative  was  eliminated  from  further  consideration 
because  it  was  very  similar  to  the  Program  Manager  Defines  alternative. 

The  Do  No  Harm  alternative  was  eliminated  because  the  team  planned  to  consider 

the  performance  of  baseline  processes  from  other  organizations  (JTEM,  MC3T, 

40 


MCTSSA)  from  the  start,  and  a  primary  goal  of  this  alternative  generation  process  is  to 
consider  two  entirely  new  alternatives,  rather  than  exclusively  baseline  processes. 

The  Capabilities  Documentation  alternative  was  eliminated  because  it  is  similar  to 
the  SAR  alternative  in  that  both  alternatives  rely  on  outside  documentation  for 
identification  of  threshold  values. 

The  revised  list  of  alternatives  was  left  as  Ad-Hoc,  SAR,  Deliberate  Method, 
Program  Manager  Direction,  and  Change  Driven.  The  team  completed  this  process  with 
a  smaller  set  of  feasible  and  sufficiently  varied  alternatives  which  could  be  analyzed  over 
the  course  of  the  project.  In  this  manner,  feasibility  screening  reduced  the  nine 
alternatives  seen  in  Table  3  to  five,  as  seen  in  Table  4.  A  detailed  description  of  the 
feasibility  screening  process  is  included  at  Appendix  D 

Because  the  FEDOS  alternative  represented  the  baseline  for  C4I  SoS  evaluation 
performance,  each  alternative  was  compared  to  the  baseline  for  each  EM.  The  team 
reviewed  the  known  performance  of  the  baseline  against  the  projected  performance  of  the 
alternative  processes.  The  scoring  criteria  were  as  follows:  (+)  the  alternative  was 
expected  to  perform  better  than  the  baseline,  (-)  the  alternative  was  expected  to  perform 
worse  than  the  baseline,  and  (=)  the  alternative  was  expected  to  perform  the  same  as  the 
baseline.  After  evaluating  each  alternative,  the  (+)  values  were  summed  and  the  (-) 
values  were  subtracted,  to  calculate  a  final  score.  Deliberate  Method  and  Change  Driven 
alternatives  were  ranked  highest,  as  displayed  in  Table  4. 
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Ad- 

Hoc 

SAR 

Deliberate 

Method 

PM 

Direction 

Change 

Driven 

Evaluation  Measures 

Percentage  of  traceable  thresholds 

= 

+ 

+ 

+ 

+ 

Percentage  of  Capabilities  Identified 

= 

+ 

+ 

+ 

= 

Time  to  Plan  Evaluation 

+ 

= 

= 

= 

+ 

Number  of  Traceable  Thresholds 
Identified 

= 

+ 

+ 

+ 

+ 

Percentage  of  Shortfalls  Identified 

= 

= 

+ 

= 

= 

Quality  of  Planning  Outputs 

= 

= 

+ 

= 

= 

Number  of  C4I  Systems  Under  Test 

= 

= 

= 

= 

= 

Percentage  of  Interfaces  Identified 

= 

= 

+ 

= 

+ 

Total  + 

1 

3 

6 

3 

4 

Total  - 

0 

0 

0 

0 

0 

Overall  Total 

1 

3 

6 

3 

4 

Table  4.  Reevaluated  JC3M  System  Alternatives  Datum  Design 
Comparison  Matrix. 

This  table  illustrates  the  results  of  the  screening  of  alternatives.  The  expected 
performance,  by  function,  of  each  of  the  potential  alternatives  solutions  was  compared  to 
the  performance  of  the  FEDOS  process.  Where  the  potential  solution  was  projected  to  be 
superior,  a  “+”  is  recorded;  where  inferior,  a  is  recorded.  Comparable  performance  is 
recorded  with  “=”,  and  the  overall  comparisons  are  summed.  “Deliberate  Methods”  and 
“Change  Driven”  had  the  highest  scores  of  the  potential  alternatives. 

1.  Alternative  Refinement 

After  identifying  the  most  promising  new  alternatives  for  consideration  as  JC3M 
solutions,  the  team  began  detailed  design  of  these  alternatives,  prior  to  developing 
simulations  to  evaluate  the  performance  of  each  alternative. 

In  referring  back  to  the  original  problem  statement,  the  team  determined  that 
threshold  performance  criteria  must  be  traceable  back  to  doctrinally  correct  sources,  or 
else  they  would  be  perceived  as  based  on  personal  opinion,  rather  than  Service  or  DoD 
requirements.  In  order  to  provide  the  most  accurate,  doctrinally  sound,  and  traceable 
performance  measures,  the  team  searched  for  doctrine  guidance,  and  found  it  in  the  form 
of  the  UJTL.  The  UJTL  is  a  “. .  .comprehensive  integrated  menu  of  functional  tasks, 
conditions,  measures,  and  criteria  supporting  all  levels  of  the  Department  of  Defense...” 
[CJCSM  3500.04D,  2005:  A-l], 

The  UJTL  provides  a  description  of  what  combatants,  combat  support  agencies, 

and  DoD  activities  must  perform.  More  importantly  to  JC3M,  the  UJTL  provides  a  list  of 
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performance  measures  and  criteria  for  use  in  an  evaluation.  There  are  also  Service- 
specific  subsets:  the  Army  Universal  Task  List  and  the  Universal  Naval  Task  List  (used 
by  the  U.S.  Navy,  U.S.  Coast  Guard,  and  U.S.  Marine  Corps). 

After  some  investigation,  the  team  found  that  doctrinal  references  provide 
measures  (speed,  accuracy,  duration,  %  of  enemy  forces  destroyed,  etc.)  but  not  the 
threshold  values.  As  an  example,  OP  3.2.6,  “Provide  Firepower  in  Support  of 
Operational  Maneuver”  is  a  task  in  the  UJTL,  and  measures  include  both  time  to  engage 
and  accuracy  of  engagement  [CJCSM  3500. 04D,  2005:  B-B-2], 

In  the  accuracy  measure,  percent  of  enemy  forces  destroyed,  delayed,  disrupted, 
or  degraded  is  evaluated  [CJCSM  3500. 04D,  2005;  B-B-2],  The  UJTL  does  not  state 
what  “percent  of  enemy  forces  destroyed”  is  sufficient  to  constitute  success.  The 
threshold  value  is  left  out  by  design,  in  order  to  allow  the  commander  the  flexibility  to  set 
a  threshold  in  accordance  with  the  mission,  the  conditions,  and  the  situation. 

Service-level  guidance  was  reviewed  as  well.  Marine  Corps  Training  and 
Readiness  Manuals  list  measures  and  criteria,  but  they  focus  on  functions  and  not 
capabilities.  As  an  example,  the  [Artillery  T&R  Manual,  2000:  3-IF-45]  describes  the 
task  of  controlling  a  battery  of  towed  artillery  pieces  moving  in  a  convoy,  and  responding 
to  an  emergency  request  for  artillery  support  by  halting  and  placing  the  guns  to  fire  on  a 
target.  "Lay  the  Battery  for  an  Emergency  Fire  Mission  (Hip  Shoot)  while  in  a  Convoy” 
is  stipulated  as  a  performance  task,  but  the  criteria  are  not  provided. 

The  team  determined  that  an  attempt  to  define  threshold  performance  values 
would  violate  the  spirit,  if  not  the  letter,  of  DoD  and  Service  level  guidance.  Further,  if 
the  JC3M  system  were  to  define  threshold  performance  values,  in  spite  of  DoD  guidance 
to  the  contrary,  the  recommended  thresholds  would  be  subjective,  and  subject  to 
immediate  challenge  by  operators,  subject  matter  experts,  acquisition  managers,  and 
other  testers.  If  the  results  of  a  test  are  perceived  to  be  subjective,  they  are  worth  very 
little. 

The  team  also  investigated  process  or  system  based  solutions  which  could 
generate  threshold  values,  but  could  not  find  any  such  solution.  There  are  tools  in 
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development  [Lockheed,  2007]  that  will  assess  conformance  to  standards  and 
interoperability  based  only  on  technical  exchange  of  information.  Tools  will  not  compare 
performance  for  end-to-end  operational  effectiveness  for  mission  accomplishment,  only 
compliance  to  a  technical  standard.  Interoperability  must  include  “. . .  both  the  technical 
exchange  of  information  and  the  end-to-end  operational  effectiveness  of  that  exchanged 
information  as  required  for  mission  accomplishment”  [CJCSI  3 170. OIF,  2007;  GL-9]. 

Based  on  the  number  of  possible  conditions,  missions,  unit  types,  and  components 
of  the  SoS,  the  team  determined  it  would  be  futile  to  try  to  create  a  tool  that  could 
account  for  all  the  combinations 

2.  JC3M  Changes 

Without  doctrinal  sources  for,  or  processes  to  generate,  threshold  criteria,  the 
team  determined  that  there  were  several  potential  changes  to  JC3M.  One 
recommendation  to  stakeholders  could  be  to  assemble  the  SoS  as  described  by 
stakeholders,  test  against  performance  measures,  then  record  the  performance.  This 
would  establish  a  baseline  of  performance  values,  so  that  when  SoS  components  were 
changed  in  the  future,  testers  could  ascribe  the  changed  results  to  changed  components. 
("With  Box  X,  missile  speed  was  620  mph;  with  Box  Y  installed  in  place  of  Box  X,  and 
everything  else  held  constant,  missile  speed  was  670  mph.")  The  before  and  after 
comparison  would  be  very  objective,  and  would  allow  stakeholders,  not  testers,  to 
determine  if  the  performance  was  acceptable.  This  is  the  methodology  used  at  Naval 
Surface  Warfare  Center  (NSWC)  Corona  Division,  and  will  be  used  by  MC3T  at 
MCTSSA  for  the  proof  of  concept  event  in  October  2007. 

Another  methodology  would  be  to  recommend  that  testers  use  doctrinal 
references  to  objectively  define  only  measures,  test  the  SoS  to  evaluate  its  performance 
on  those  measures,  and  report  the  results  to  the  stakeholders  ("87%  of  enemy  forces  were 
destroyed").  This  approach  depends  on  the  ability  to  assess  combat-focused  performance 
on  measures  that  are  not  typically  available  in  a  test  organization  for  space,  time,  and 
safety  constraints.  While  simulations  can  be  used  to  accomplish  this  approach,  this  adds 
another  layer  of  complexity  to  the  conduct  of  a  C4I  SoS  evaluation,  as  well  as  the 
potential  for  challenges  to  the  evaluation  methodology,  based  on  the  implementation  of 
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the  combat  simulations.  This  approach  was  considered,  but  because  JC3M  is  focused  on 
objective  measures,  it  was  not  considered. 

Another  methodology  would  be  to  recommend  further  emphasis  on  the  JCIDS, 
with  all  future  acquisitions  made  on  a  SoS  level,  with  objective  performance  criteria  at  an 
operational  level  identified  in  JCIDS  documents  before  acquisition.  This  is,  in  the  long 
run,  the  best  answer  for  DoD,  but  does  not  address  the  many  legacy  C4I  systems  that  are 
assembled,  after  the  fact,  into  a  SoS.  This  is  also  in  line  with  the  concept  of  capability 
portfolio  management  ordered  by  Deputy  Secretary  of  Defense  Gordon  England  in  2006. 
In  a  memo  defining  capability  portfolio  management  roles  and  responsibilities,  Secretary 
England  stated  the  intent  is  to  “. . .  manage  groups  of  like  capabilities  across  the  enterprise 
to  improve  interoperability,  minimize  capability  redundancies  and  gaps,  and  maximize 
capability  effectiveness”  [England,  2006:  3],  The  memorandum  also  charged  the  Joint 
C2  Capability  Portfolio  manager  to  create  a  test  case  that  “. .  .delivers  integrated  joint 
capabilities,  which  improve  interoperability,  minimize  capability  redundancies  and  gaps, 
and  maximize  joint  operational  effectiveness”  [England,  2006:  B-l],  The  Joint  C2 
Capability  Portfolio  manager,  with  three  other  portfolio  managers,  worked  on  the  2008 
DoD  Budget  request,  which  started  the  process  of  adjusting  Service  C2  programs 
[Federal  Computer  Week,  2007].  While  an  excellent  long-term  solution,  this  does  not 
address  the  immediate  need  for  performance  measures  to  use  in  evaluating  legacy  C4I 
SoS. 

The  team  chose  to  use  JC3M  to  define  performance  measures  only,  and  use  those 
measures  to  test  the  C4I  SoS.  This  represents  a  change  to  the  JC3M  effective  need,  i.e. 
instead  of  pursuing  threshold  performance  values  (“60  mph  is  the  passing  criteria”), 
JC3M  will  define  the  performance  measures  to  be  used  in  the  evaluation  of  a  C4I  SoS 
(“measure  the  speed  of  the  vehicle”). 

The  team  reviewed  the  generated  alternatives  and  determined  they  would  have 
been  effective  when  the  original  goal  of  JC3M  was  to  provide  a  method  for  organizations 
to  define  threshold  performance  criteria  for  use  in  C4I  SoS  evaluations. 
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When  documenting  the  processes  that  comprise  each  of  the  alternatives,  prior  to 
creating  models,  the  modeling  and  simulation  team  determined  that  in  defining  C4I  SoS 
performance  measures,  both  Deliberate  and  Change-Driven  alternatives  performed  the 
same  functions,  in  the  same  manner.  As  originally  defined,  each  model  took  the  same 
external  data  sources  (the  UJTL,  Systems  Engineering  Artifacts,  and  system-level 
documentation)  into  consideration  when  defining  performance  measures.  The 
alternatives  differed  in  how  they  performed  the  next  step,  i.e.  identifying  the  threshold 
values  which  were  the  goal  of  the  original  JC3M  solution.  Because  those  threshold 
values  were  no  longer  pursued,  the  final  and  discriminating  phase  of  each  alternative  was 
eliminated.  With  that  elimination,  the  modeling  and  simulation  team  determined  that  PM 
Direction  and  SAR  would  perform  exactly  alike,  thus  eliminating  their  value  in 
presenting  different  approaches  to  the  JC3M  solution. 

The  team  generated  two  new  alternatives  to  meet  the  revised  JC3M  requirements, 
utilizing  the  same  functions  that  drove  the  first  round  of  alternative  generation,  again 
using  the  “Morphological  Box  Process”  described  in  Chapter  III  section  A.3. 
a.  Systems  Capabilities  Review  Alternative 
The  Systems  Capabilities  Review  (SCR)  alternative  is  a  combination  of 
the  “Capabilities  Documentation”  alternative  and  the  “Test  Agency  Direction”  alternative 
outlined  in  Table  3.  SCR  is  composed  of  a  group  of  stakeholders:  C4I  system  and  SoS 
user  representatives,  test  agency  representatives,  system  developers,  system  designers, 
and  program  managers.  The  test  agency  representative  chairs  the  group,  which  meets  as 
required  to  support  a  C4I  SoS  evaluation,  at  the  Systems  Command  level. 

Inputs  to  SCR  include  source  documents  such  as  the  Capabilities 
Development  Document  (CDD),  Operational  Requirements  Document,  Test  Plans, 
TEMPs,  Concept  of  Operations  documents,  Joint  Integrating  Concepts,  Joint  Operating 
Concepts,  and  system  level  metrics. 

The  SCR  first  reviews  SoS  capabilities  specifications,  and  will  examine 
the  systems  engineering  artifacts  already  created,  such  as  supporting  DoD  Architecture 
Framework  documents  and  technical  performance  measures,  and  creates  a  list  of  implied 

and  stated  SoS  capabilities.  Next,  the  SCR  reviews  system-level  documents  and  creates  a 
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system-level  capabilities  list.  Third,  the  SCR  maps  system-level  capabilities  to  SoS  EM. 
The  SCR  identifies  gaps  in  the  EM  list,  and  creates  the  balance  of  the  EM  necessary  to 
evaluate  the  performance  of  the  C4I  SoS.  Figure  15  illustrates  how  SCR  performs  the 
JC3M  subfunction  1.3.2  “Define  Measures.” 


SCR  Alternative 


Figure  15.  List  of  Sub  Tasks  Performed  by  SCR  to  Define  Measures. 

The  JC3M  Functional  Hierarchy  calls  for  function  1.3  “Define  Evaluation  Criteria.”  This 
function  is  supported  by  JC3M  subfunction  1.3.2  “Define  Measures.”  The  SCR 
alternative,  in  order  to  support  1 .3.2,  performs  sub  processes.  Each  of  the  subprocesses 
may  have  multiple  submissions  from  C4I  SoS  evaluation  stakeholders. 
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The  subtasks  performed  by  SCR  as  part  of  Task  1.3.2  “Define  Measures” 


are  illustrated  in  more  detail  in  Figure  16. 


Figure  16.  SCR  Define  Measures  Subtask  Ordering. 

This  figure  illustrates  the  subtasks  broken  down  in  the  order  in  which  they  are  performed 
in  the  SCR  alternative.  Review  SoS  Capabilities,  Review  System  Level  Documents,  and 
Define  Measurement  Conditions  and  Rules  can  be  worked  in  parallel.  Once  these  tasks 
are  complete  the  team  maps  EM  SoS  capabilities  and  reviews  gap  areas.  These  functions 
are  consistent  with  the  more  detailed  functional  analysis  performed  on  the  new 
alternatives.  “REF”  is  a  CORE  artifact,  and  indicates  no  additional  function  is  enabled 
when  this  flow  completes. 

b.  Functional  Capabilities  Board  Alternative 
The  Functional  Capabilities  Board  (FCB)  alternative  is  based  on  an 
existing  group,  the  JCIDS  C2  Functional  Capabilities  Board,  to  define  the  performance 
measures  of  the  C4I  SoS.  Figure  17  illustrates  the  tasks  of  an  FCB,  which  is  to  perform 
“. . .  organization,  analysis,  and  prioritization  of  joint  warfighting  capabilities  within  an 
assigned  functional  area”  [CJCSI  3170. OIF,  2007;  GL-7].  Inputs  to  the  FCB  include  the 
UJTL  and  subsets,  Concept  of  Operations  (CONOPS)  documentation,  Program  Change 
Request  documentation,  and  system  trouble  reports. 


Standing  FCBs  evaluate  shortfalls  and  gaps  in  current  and  future 
capabilities  as  they  are  identified  by  studies  or  assessments,  or  arise  from  processes  or 
meetings  [CJCSI  3137. 01C,  2004:  E-l],  The  FCB  reports  are  provided  to  the  Joint 
Capabilities  Board  (JCB)  or  Joint  Requirements  Oversight  Council  (JROC).  The 
additional  effort  proposed  in  this  alternative  represents  an  increase  in  the  work  performed 
by  the  C2  FCB,  but  is  in  the  same  functional  area,  and  engages  in  many  of  the  same 
tasks.  Unlike  the  SCR,  the  FCB  meets  on  an  ongoing  basis,  rather  than  as  required  to 
support  C4I  SoS  evaluations. 
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The  FCB  will  first  identify  the  configuration  of  the  SoS  by  determining 
the  component  systems.  Next,  the  FCB  will  identify  the  SoS  capabilities.  SoS  CONOPS 
are  reviewed  to  determine  EM,  and  finally  the  FCB  will  generate  the  SoS  EM  list  for  use 
in  C4I  SoS  evaluations.  The  FCB,  under  JCIDS,  has  a  long-term  perspective  and  charter. 
The  FCB  represents  a  near-term  resource  that  can  provide  ongoing  analysis,  and  so 
reduce  the  lack  of  C4I  SoS  performance  measures.  The  FCB  appears  to  explicitly  support 
the  concept  of  capability  portfolio  management,  but  the  JC3M  team  could  not  find  any 
information  to  confirm  that  relationship. 
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Figure  17.  JCIDSandFCB. 

FCB  is  central  to  JCIDS  and  provides  Service  and  Functional  Area  representation  as  well  as  analysis  of  existing  capabilities  and  future 
shortfalls  in  capabilities.  FCB  analysis  is  provided  to  the  Joint  Capabilities  Board  and  Joint  Requirements  Oversight  Council  [From  CJCSI 
3137. 01C,  2004]. 
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The  external  agency  relationship  between  the  FCB  and  C4I  test 
organizations,  and  the  list  of  sub  tasks  needed  to  complete  the  Define  Measures  task,  is 


illustrated  in  Figure  18. 


Figure  18.  List  of  Sub  Tasks  needed  to  Complete  Define  Measures. 

The  FCB  Alternative  performs  the  task  of  defining  measures,  using  a  team  external  to  the 
test  organization  that  meets  weekly  throughout  the  year.  Once  the  test  agency  enters  their 
planning  process  for  a  C4I  Evaluation,  they  elicit  input  from  the  FCB  team  on  capability 
measures.  These  measures  are  analyzed  for  feasibility  and  testability  before  they  are 
included  in  data  collection  and  analysis. 

Because  the  FCB  is  external  to  the  test  organization,  some  analysis  of  the 
performance  measures  generated  by  the  FCB  will  be  necessary.  Figure  19  illustrates  both 
the  external  nature  of  performance  measure  generation,  for  this  alternative,  as  well  as  the 
required  analysis  performed  internally  to  the  test  organization. 
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Figure  19.  FCB  provides  external  input  to  C4I  Test  Organization. 

This  illustration  highlights  that  the  FCB  is  external  to  the  Test  organization,  and  both 
organizations  continuously  coordinate  their  efforts.  The  FCB  meets  and  generates 
performance  measures  independent  of  the  test  organization’s  schedule.  From  a  planning 
perceptive;  the  test  organization  need  only  analyze  incoming  measures  for  feasibility  and 
testability. 


c.  Differences  between  SCR  and  FCB 

From  the  JC3M  Functional  Hierarchy  perspective,  SCR  and  FCB  appear 
the  same.  The  difference  between  these  alternatives  is  in  the  approach  each  alternative 
takes  to  complete  process  1.3.2  “Define  Measures”  in  the  JC3M  Functional  Hierarchy,  as 
illustrated  in  Figure  20. 
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SCR  incorporates 
all  the  tasks 
associated  with  this 
process  into  the  test 
planning  process 


FCB  utilizes  an 
external  Team  that 
meets  year  round  to 
provide  Capability 
Measures  to  the  test 
agency 


Figure  20.  SCR  and  FCB  Variance. 

The  FCB  and  SCR  alternatives  differ  in  subtasks  and  their  approach  to  JC3M  function 
1.3.2  “Define  Measures.”  SCR  performs  this  task  with  internal  resources,  and  in 
response  to  requests  for  C4I  SoS  evaluation.  FCB  is  a  resource  external  to  the  C4I  test 
organization,  and  performs  this  task  weekly. 

The  alternatives  differ  in  the  source  of  personnel  required  to  execute  a  C4I  SoS 
evaluation;  whether  they  have  been  used  in  a  historical  event,  have  been  simulated,  or  are 
a  new  proposal;  what  organization  chartered  their  development  and  execution;  and  the 
sources  used  to  define  performance  measures.  Table  5  summarizes  the  key  similarities 
and  differences  between  the  alternatives. 
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Personnel 

Resources 

Used  for 
Testing 

Service  or  Joint 

Performance 
Measure  Creation 

FEDOS 

Internal 

Yes 

Executed  by 
Service  test 
organization, 
driven  by  Service 
stakeholders 

Stakeholder 

agreement 

MC3T 

Internal 

Yes1 

Service  level 
charter  for 
certification  of 
C4I  capabilities 

Doctrine 
Developers  and 
C4I  system 
stakeholders 

JTEM 

CTM 

Internal 

No2 

Developed  for 
joint  use 

Review  of 
doctrine,  system 
documentation 

FCB 

Internal  / 
External 

No 

Proposed  for  joint 
use 

Review  of  Doctrine 
and  system 
documentation  by 
Joint  C4I  SME 
Panel 

SCR 

Internal 

No 

Proposed  for  joint 
use 

Doctrine 
Developers  and 
C4I  system 
stakeholders 

Table  5.  Summary  Comparison  of  Alternatives. 

The  table  summarizes  key  features  of  the  alternatives  under  consideration.  Resources 
indicate  if  personnel  used  in  the  conduct  of  a  C4I  SoS  evaluation  are  internal  or  external 
to  the  test  organization.  Used  for  Testing  indicates  if  the  alternative  has  been  used  at  a 
test  organization.  'MC'3T  was  projected  to  execute  a  proof  of  concept  in  October  2007. 

2JTEM  CTM  has  executed  simulations,  and  was  projected  to  conduct  limited-scope  tests 
in  late  2007  (results  not  available  for  review).  Origin  indicates  the  organization  that 
ordered  the  development  or  execution  of  the  alternative.  JTEM  CTM  is  a  Joint  Test  and 
Evaluation  project  chartered  by  the  Director  of  Operational  Test  and  Evaluation.  Source 
of  Performance  Measures  indicates  where  performance  measures  are  generated. 

C.  ALTERNATIVE  SCORING  CRITERIA 

After  the  revisions  to  the  JC3M  approach,  revised  EM  were  required  to  rank  the 
alternatives.  The  original  nine  EM  created  as  part  of  the  JC3M  value  hierarchy,  and 
documented  in  Appendix  D,  were  reviewed,  and  those  specific  to  threshold  values 
(Percentage  of  Traceable  Thresholds,  Number  of  Traceable  Thresholds  Identified)  were 
eliminated.  EM  that  measured  the  SoS  configuration  (Percentage  of  Capabilities 
Identified,  Percentage  of  Shortfalls  Identified,  Number  of  C4I  SUT,  Percentage  of 
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Interfaces  Identified),  rather  than  measuring  the  production  of  performance  measures, 
were  eliminated. 

Those  EM  that  measured  how  well  the  alternative  performed  (Percentage  of 
Traceable  Measures,  Quality  of  Planning  Outputs)  were  retained,  as  well  as  those  EM 
that  measured  the  duration  of  the  alternative  process  (Days  to  Plan  Evaluation).  Labor 
Hours  were  retained,  and  though  not  seen  below,  were  used  in  the  Life  Cycle  Cost 
Estimate  (LCCE)  for  each  alternative.  Finally,  those  EM  that  measured  the  flexibility  of 
the  alternative  (Elasticity  of  Labor,  Elasticity  of  Duration)  were  also  retained.  Table  6 
lists  the  revised  EM  used  to  rank  the  performance  of  each  alternative. 


Percentage 
of  Traceable 
Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

1.3.2 

1.4.3 

1.4.3 

4.1 

4.1 

FEDOS 

MC3T 

ITEM  CTM 

FCB 

SCR 

Table  6.  JC3M  Scoring  Matrix. 

EMs  used  for  comparing  alternatives  measure  traceability  of  outputs,  duration,  quality  of 
outputs,  and  the  response  of  the  alternative  to  changes  in  SoS  size.  Labor  hours  were  later 
recorded,  but  used  in  the  LCCE,  required  for  the  final  Cost/Benefit  analysis. 
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IV.  MODELING  AND  SIMULATION 


A.  JC3M  USES  OF  MODELING  AND  SIMULATION 

Modeling  and  Simulation  (M&S)  was  the  most  important  element  of  the  JC3M 
SE  process,  because  it  allowed  alternative  solutions  to  be  compared  in  an  affordable, 
effective,  and  repeatable  manner.  If  real  systems  could  be  utilized  in  their  operating 
environment,  comparisons  would  be  of  higher  fidelity,  but  this  is  not  often  possible.  In 
many  situations,  safety,  practicality,  or  resource  constraints  make  it  impossible  to  utilize 
alternatives  in  their  proposed  environment.  It  is  not  acceptable,  for  example,  to  provoke 
a  war  to  test  the  effectiveness  of  a  defensive  system.  It  is  also  not  prudent  to  purchase 
and  field  a  system,  without  an  acceptable  estimate  of  performance,  before  the  start  of 
hostilities.  The  team  could  not  build  facilities  and  implement  each  alternative  process 
for  C4I  SoS  evaluation  in  order  to  compare  their  effectiveness  and  measure  their  costs,  so 
M&S  was  used. 

A  model  is  a  physical,  visual,  or  mathematical  representation  of  a  real  system  that 
accounts  for  its  known  or  inferred  properties  and  may  be  used  for  further  study  of  its 
characteristics  [Forsberg,  Mooz,  Cotterman,  2000:  18].  The  purpose  of  a  mathematical 
model  is  to  represent  key  characteristics  of  the  real  system  that  we  want  to  study,  such  as 
operating  cost,  productivity,  or  time  to  market.  The  purpose  of  a  visual  model  is  to  allow 
analysis  of  key  relationships,  inputs  and  outputs,  and  functional  flows  of  the  real  system. 
Software  tools  were  used  to  build  both  mathematical  and  visual  models  of  each 
alternative. 

Simulation  operates  the  model  with  selected  inputs,  outputs,  and  environmental 
conditions  in  order  to  analyze  the  performance  of  the  system.  Models  of  each  alternative 
were  generated,  and  simulations  were  executed  on  the  models  utilizing  different  data 
inputs  in  a  systematic  and  logical  manner  to  gain  insight  into  how  each  alternative  would 
perform.  The  outputs  from  the  models  and  simulators  were  analyzed  by  the  team,  and  the 
results  provided  input  into  the  Analysis  of  Alternatives  phase.  With  the  exception  of  the 
FEDOS  alternative,  none  of  the  alternatives  under  consideration  have  historical 
performance  data  available  for  analysis;  therefore  modeling  and  simulation  was  an 
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effective  method  to  analyze  the  alternatives,  predict  behavior  under  varying  conditions, 
and  compare  the  performance  of  the  different  alternatives. 

Figure  21  identifies  two  of  the  modeling  and  simulations  used  to  generate  three  of 
the  five  EM  used  to  compare  the  performance  of  the  alternatives.  POW-ER  (Project, 
Organization,  and  Work  for  Edge  Research)  refers  to  a  modeling  and  simulation  tool 
developed  by  Stanford  University  to  optimize  the  work  flow  of  an  organization.  Arena 
refers  to  the  Rockwell  Automation  tool  that  can  model  how  a  system  behaves  over  time. 
POW-ER,  Arena,  and  CORE  (another  M&S  tool)  are  described  in  detail  in  Section  B 
“JC3M  Modeling  Tools.”  In  Figure  21,  “offline  evaluation”  refers  to  the  consultation 
with  SMEs  (described  in  Appendix  C)  that  was  used  to  generate  the  remaining  EM. 


Figure  21.  M&S  Contribution  to  JC3M  Scoring  Matrix. 

This  figure  illustrates  the  three  EM  which  relied  on  M&S  for  data.  The  “offline 
evaluation”  in  the  figure  refers  to  the  SME  consultation  described  in  Appendix  C. 

M&S  enabled  the  team  to  predict  the  behavior  of  an  alternative  when  input 

parameters  changed.  The  team  was  able  to  identify  what  would  happen  if  the  input 
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parameters  to  a  system  increased  by  10%.  Would  the  labor  hours  and  calendar  duration 
increase  by  the  same  amount?  It  was  important  to  be  able  to  answer  these  questions, 
because  they  provided  insight  into  the  behavior  of  the  alternatives.  The  effects  on 
duration  and  labor  hours  related  directly  to  the  elasticity  of  the  alternative. 

B.  JC3M  MODELING  TOOLS 

The  M&S  tools  used  in  the  project  allowed  the  team  to  generate  models  and  run 
simulations  to  analyze  the  required  EM  for  each  of  the  JC3M  alternatives. 

1.  CORE 

CORE  is  a  systems  engineering  software  tool  that  provided  traceability  from 
Problem  Refinement  through  AoA.  The  team  used  CORE  to  construct  standard 
Integration  Definition  for  Function  Modeling  (IDEFO)  diagrams.  The  IDEFO  diagrams 
provided  the  reference  architecture  for  analysis  of  the  alternative  systems.  The  IDEFO 
diagrams  allowed  the  team  to  identify  and  analyze  the  functions  that  are  performed  by 
JC3M,  as  well  as  by  each  alternative,  and  to  record  the  mechanisms  (means)  by  which 
they  were  performed.  The  graphical  representation  of  the  systems  provided  by  CORE 
through  the  IDEFO  permitted  the  team  to  conceptualize  the  functional  requirements  of  the 
systems.  IDEFO  diagrams  helped  to  facilitate  understanding,  communication,  consensus 
and  validation  of  the  systems  under  study  [Federal  Information  Processing  Standards 
Publication,  2007:  183],  Samples  of  the  CORE  IDEFO  analyses  are  provided  in 
Appendix  F. 

2.  POW-ER 

POW-ER  (Project,  Organization,  and  Work  for  Edge  Research)  is  a  project 
organization  modeling  and  simulation  tool  that  integrates  organizational  and  process 
views.  POW-ER  is  developed  as  an  outgrowth  of  the  Virtual  Design  Team  (VDT) 
computational  modeling  research  at  Stanford  University.  POW-ER  addresses  the 
organizational  elements  that  impact  the  ability  to  work  effectively,  including:  policies 
and  structures  (culture,  communication,  decisions,  meetings);  staffing,  hiring,  and 
training  needs  for  workforce  plans.  Using  POW-ER,  the  team  modeled  the 
organizational  structure,  the  relationship  between  individuals  within  those  organizations, 
and  individual  task  allocations. 
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The  Days  to  Plan  Evaluation  EM,  identified  in  the  M&S  Contribution  to  JC3M 
Scoring  Matrix  (see  Figure  21),  is  directly  related  to  the  management  and  organization 
aspect  of  the  JC3M  system.  POW-ER  allowed  experimentation  with  “What-If  ’  scenarios 
and  provided  insight  into  the  level  of  rework,  backlog,  and  risk  associated  with  an 
alternative.  POW-ER’ s  ability  to  predict  and  analyze  backlogs  proved  to  be  quite  useful 
in  the  design  and  troubleshooting  of  alternative  models,  because  it  allowed  the  team  to 
identify  backlogs  in  the  workflow  of  models.  The  analysis  of  backlogs  in  the  workflow 
enabled  the  team  to  identify  the  optimized  arrangement  of  tasks  and  personnel  for  both 
the  SCR  and  FCB  alternatives. 

Each  POW-ER  model  was  generated  by  adding  the  tasks  and  personnel  identified 
for  an  alternative  from  their  respective  data  sets  (details  provided  later  in  this  chapter) 
Tasking  flow  was  modeled  based  on  the  defined  relationship  from  the  data  sets  (i.e., 
parallel  tasks  were  modeled  to  occur  simultaneously  and  serial  tasks  were  modeled  to 
occur  based  on  dependencies).  Next,  personnel  that  perform  similar  tasks  were  assigned 
into  a  team;  those  teams  were  then  linked  to  a  respective  task. 

Both  parallel  and  serial  task  relationships  can  be  seen  in  Figure  22,  which  depicts 
examples  of  each  of  the  top  level  functions  (Elicit  Requirements,  Generate  Evaluation 
Plan,  Develop  Test  Plans  and  Procedures)  of  the  FEDOS  process. 
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Figure  22.  POW-ER  Baseline  Model. 

Parallel  and  Serial  tasks  are  modeled  in  POW-ER,  based  on  historical  performance 

(FEDOS),  stakeholder  input  (MC3T,  JTEM  CTM)  or  design  of  the  alternative  (FCB, 

SCR).  Models  contain  both  tasks  and  personnel  assigned  to  the  tasks. 

Within  POW-ER,  the  Effort  attribute  for  each  task  was  set  by  assigning  it  to  the 
duration  in  days  identified  in  an  alternative’s  data  set.  For  the  Team  attribute,  the  Full 
Time  Equivalent  (FTE)  attribute  was  set  equal  to  the  total  hours  an  alternative  team  spent 
on  each  task  divided  by  40  hours  (one  work  week). 

It  is  important  to  mention  that  the  POW-ER  simulation  tool  calculates  a  task’s 
duration  in  one  of  two  ways:  simulated  duration  or  Critical  Path  Method  (CPM). 
Simulated  duration  factors  in  the  “hidden  works”  that  the  CPM  does  not.  The  “hidden 
works”  associates  an  amount  of  rework  and  delays  into  each  task.  The  simulated 
duration  was  selected  for  Days  to  Plan  Evaluation  EM,  because  it  is  fair  to  assume  that 
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these  hidden  works  are  always  part  of  each  task.  The  accuracy  of  the  POW-ER  model 
was  validated  against  historical  performance  data  (details  provided  later  in  this  chapter). 

3.  ARENA 

Arena  provided  a  numerical  evaluation  of  a  system  by  imitating  the  system’s 
operations  or  characteristics  over  time.  Arena  allowed  the  team  to  conduct  numerical 
experiments  in  order  to  predict  the  behavior  of  an  alternative  given  a  set  of  conditions. 
The  JC3M  Scoring  Matrix  identifies  two  EM  (see  Figure  21)  that  required  evaluation  of 
the  changes  in  output  as  a  function  of  the  changes  in  inputs:  Elasticity  of  Labor  and 
Elasticity  of  Duration.  Arena  allowed  the  team  to  run  simulations  on  the  alternative 
models  with  varying  set  of  inputs;  Arena  displayed  the  output  changes  that  corresponded 
to  each  of  the  varying  inputs. 

Figure  23  shows  results  of  a  simulation  run  in  Arena  that  predicts  the  duration  and 
labor  hours  require  for  planning  a  SoS  evaluation.  As  noted,  controls  can  be  varied. 


Input  Variables  can  be 
altered  to  collected  the 
simulated  responses 


Duration 


Figure  23.  Arena  Simulation  Run. 

Simulation  allowed  the  team  to  change  input  parameters  to  reflect  the  concerns  of 
stakeholders.  Outputs  were  measured,  in  response  to  input  changes,  to  analyze  the 
performance  of  the  alternatives  in  changing  situations. 
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The  team  created  a  baseline  C4I  SoS  to  use  in  comparing  the  performance  of  each 
alternative  in  Arena  (See  Section  C  “Select  a  Baseline”  and  Section  D.l  “Baseline  C4I 
Architecture”  for  details  of  the  baseline  C4I  SoS).  After  the  models  of  alternatives  were 
created  (see  Section  C.7  “Build  POW-ER,  Arena,  and  CORE  Model  for  each 
Alternative”),  Arena  was  used  to  calculate  the  duration  of  each  task.  Systematic 
variations  to  the  inputs  (number  of  individual  systems  and  number  of  capabilities)  were 
made  to  analyze  how  changes  in  the  size  of  the  C4I  SoS  affected  the  EM  of  interest. 

From  the  varied  inputs,  and  resulting  outputs,  the  elasticity  EM  needed  for  the  AoA  was 
calculated. 

All  Arena  models  are  based  on  data  sets  containing  process  tasks,  job  billets,  and 
hours  worked.  For  process  tasks  that  contained  sub-processes,  the  team  created  sub 
models  as  illustrated  in  Figure  24. 
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Baseline  Planning  Process  Model 

To  validate  tins  inodd  with  r«l-w<wld  Man  Imcs .  use  19  Systems,  4  New  capabilities,  and  ID  Old  Capabilities 
Output  needs  to  be  rerthh  5*fo  of:  6,4S2  T PtafTrne  ri  Man  liars 


Figure  24.  Example  of  Arena  Sub-Models  and  Sub-Processes. 

Each  alternative’s  Arena  model  had  sub-models  that  contained  a  series  of  process  delays 
that  affected  Duration  and  Total  Labor  hours.  The  number  of  sub-models  and  associated 
tasks  depended  on  the  design  of  the  alternative. 

The  sub  models  used  in  Arena  contain  process  delays,  as  appropriate,  which  were 
used  to  represent  average  labor  hours  per  variable  instance.  Figure  25  illustrates 
variables,  units,  and  the  mathematical  expression  used  in  calculating  an  example  process 
delay. 
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“OldCaps”,  “NewCaps”,  and  “SysNum”  are  variables 
that  are  manipulated  by  the  Process  Analyzer  to 
estimate  behavior.  Each  variable  is  multiplied  by  a 
calculated  value  assigned  to  it  for  each  task.  These 
values  represent  average  hours  spent  on  each 
variable’s  item. 
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Figure  25.  Example  of  a  Process  Delay  Setting. 

Each  sub  model  contained  a  series  of  process  delays  used  to  represent  labor  hours  per 
variable  instance.  The  variables  identified  ( OldCaps ,  NewCaps,  and  SysNum)  were  called 
and  varied  in  a  centralized  process  analyzer  which  enabled  the  modeling  team  to  run 
multiple  variations  of  each  alternative. 

Each  alternative  was  modeled  in  Arena,  and  imported  into  the  Arena  Process 
Analyzer  for  efficient  simulation  of  the  alternative  using  varied  inputs.  The  team  utilized 
the  control  feature  to  centrally  control  and  vary  input  variables  to  study  the  associated 
variability  of  outputs.  Each  model  used  three  critical  input  parameters  to  determine  the 
size  and  the  scale  of  the  test  event.  These  parameters  are: 

■  Number  of  Systems  (SysNum}  -  This  parameter  did  not  represent  the 
number  of  instances  of  a  system  (i.e.,  14  AFATDS  terminals),  but 
represented  the  number  of  types  of  systems  under  test.  If  the  SoS 
contained  PLI  systems,  communication  systems,  and  fire  control  systems, 
SysNum  equaled  3. 

■  Number  of  New  Capabilities  f NewCaps )  -  This  parameter  represented  the 
number  of  SoS  capabilities  under  evaluation.  If  the  C4I  test  organization 
had  little  to  no  experience  with  this  capability,  it  was  considered  a  new 
capability. 
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■  Number  of  Old  Capabilities  (OldCaps)  -  This  parameter  also  represented 
the  number  of  SoS  capabilities  under  evaluation.  If  the  C4I  test 
organization  was  familiar  with  and  had  tested  the  capability,  it  was 
considered  an  Old  Capability.  This  parameter  addressed  the  concept  of 
efficiency  associated  with  experience. 

Figure  26  shows  a  view  of  the  Process  Analyzer  tool,  and  illustrates  the  multiple 
versions  of  each  alternative  and  the  varying  magnitudes  of  NewCaps,  OldCaps,  and 
SysNum. 
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Figure  26.  Screen  Shot  from  the  Arena  Process  Analyzer. 

Arena  Process  Analyzer  is  used  to  centrally  vary  input  variables  for  each  alternative 
model  in  order  to  examine  the  effects  on  the  performance  of  the  alternative.  In  this 
illustration,  the  alternatives  are  identified  in  the  left  column,  with  sizes  varied  by  25% 
steps.  This  data  is  used,  in  turn,  to  conduct  analysis  of  the  performance  of  each 
alternative. 


66 


C.  MODELING  AND  SIMULATION  APPROACH 

The  JC3M  team  implemented  an  eight  step  M&S  approach.  The  details  of  this 
approach  are  provided  for  clarification. 

1.  Select  a  Baseline  System 

A  baseline  of  system  performance  was  selected  that  could  be  used  to  validate  the 
models  created  by  the  team.  The  team  chose  the  FEDOS  alternative  for  designing  and 
validating  performance  models  of  all  alternatives.  Use  of  FEDOS  allowed  the  team  to 
compare  the  performance  from  M&S  tools  to  the  historical  data  documenting  the  actual 
performance  of  FEDOS,  thus  validating  the  accurate  performance  of  the  model. 

Inputs  such  as  number  of  systems,  capabilities,  and  tasks  were  gathered  from  the 
historical  record,  models  were  designed,  and  simulation  outputs  were  generated.  Outputs 
were  compared  to  actual  results,  and  the  model  was  systematically  adjusted  until  it 
produced  results  that  were  close  to  actual  labor  hours  and  duration  (Days  to  Plan 
evaluation)  of  the  historical  event.  The  labor  hour  calculations  were  validated  to  within 
±1%  of  historical  values  because  this  data  was  used  was  used  to  calculate  the  LCCE  of 
the  alternatives.  Duration  was  validated  to  within  ±4%  of  historical  values  because  this 
EM  only  contributed  5.8%  to  the  overall  performance  score  (see  Chapter  VI,  Section  E 
“Analytical  Hierarchy  Process”)  of  the  alternative.  This  provided  the  team  confidence 
that  the  models  built  in  Arena  and  POW-ER  accurately  emulated  the  performance  of  the 
FEDOS  alternative,  and  could  be  used  to  accurately  simulate  the  performance  of  other 
alternatives. 

As  noted  earlier,  each  of  the  five  alternatives  focused  on  the  planning  of  a  C4I 
SoS  evaluation.  Figure  27  illustrates  the  Planning  function  of  JC3M. 
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Figure  27.  JC3M  Planning  Functions. 

The  models  of  alternatives  were  validated  by  comparison  to  FEDOS  historical 
performance.  After  adjustment  of  the  models  to  produce  results  within  1%  of  historical 
FEDOS  performance,  each  of  the  alternatives  was  modeled  in  turn.  This  illustrates  the 
entire  planning  functional  hierarchy  of  JC3M;  Figure  1 1  only  illustrates  the  critical  EM. 

2.  Select  a  Candidate  C4I  Test  Architecture  as  a  Baseline  SoS 

The  JC3M  team  selected  the  C4I  SoS  architecture  tested  in  2005,  using  the 
FEDOS  process,  as  stated  above,  because  historical  performance  data  existed  which 
allowed  consistent  comparison  of  the  models  to  actual  results.  The  architecture  that  was 
the  subject  of  the  FEDOS  2005  Event  was  composed  of  nineteen  different  types  of 
systems  (described  in  Section  D  “Baseline  Model  Development”  of  this  chapter).  The  test 
evaluated  fourteen  SoS  capabilities;  ten  were  previously  tested  (“old”)  and  four  were  new 
to  the  test  team  (“new”).  Each  of  the  alternatives  was  modeled  with  this  architecture  as 
the  C4I  SoS  under  evaluation. 

This  consistency  of  C4I  SoS  under  evaluation  is  important  because  the  FEDOS 
2005  was  a  real  SoS  evaluation  event  that  could  be  modeled  and  simulated.  The  results 
of  simulation  could  be  compared  to  historical  performance  data.  Further,  establishment 
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of  a  baseline  C4I  SoS  allowed  for  an  “apples  to  apples”  comparison  of  the  performance 
of  each  alternative,  as  they  were  used  to  plan  the  evaluation  of  the  same  C4I  SoS. 

3.  Assemble  a  Data  Set  for  the  FEDOS  System 

A  data  set  is  the  compilation  of  data  (duration  of  tasks,  resources,  sequence  of 
events,  labor  hours)  used  to  populate  the  model.  In  this  case,  historical  documents  that 
recorded  the  performance  of  FEDOS  2005  were  used  as  the  sources  of  data  that 
documented  the  configuration  of  the  baseline  system.  Three  historical  documents  were 
used  to  build  the  baseline  data  set. 

The  FEDOS  2005  Technical  Support  Plan  (TSP)  was  a  Microsoft  Project®  plan 
that  specified  the  tasks  performed  and  the  order  in  which  they  were  performed;  identified 
the  resources  used  for  each  task;  and  identified  serial  task  dependencies  and  parallel 
tasks  [Manning  (b),  2005].  The  Labor  Hours  report  [Manning  (a),  2005]  recorded  the 
hours  worked  by  each  employee  for  each  task.  The  Test  Report  [Marine  Corps  Tactical 
Systems  Support  Activity,  2005]  provided  the  evaluation  of  the  conduct  of  FEDOS  in 
2005.  Figure  28  provides  an  illustration  of  the  process  of  conversion  of  historical  data 
into  the  baseline  data  set. 


Figure  28.  JC3M  Baseline  Data  Set  Inputs  and  Outputs. 

The  baseline  data  set  was  built  from  three  historical  records,  and  was  used  to  populate  the 
models  of  each  alternative  in  turn.  This  allowed  validation  of  the  models  against 
historical  data,  as  well  as  evaluation  of  the  performance  of  each  alternative  against  the 
same  size  and  scope  C4I  SoS. 
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Data  from  the  TSP,  Labor  Hours  Report,  and  Test  Report  were  summarized  in  a 
spreadsheet  that  consisted  of  the  following  list  of  data  for  use  by  the  models  of  each 
alternative: 

■  a  comprehensive  list  of  planning  tasks  used  for  test  planning, 

■  labor  hours  for  each  test  team  member  (per  task), 

■  calendar  duration  for  the  planning  process, 

■  inputs  and  outputs  for  each  task, 

■  each  test  team  member’s  cost  per  hour, 

■  test  team  position  such  as  Test  Lead,  or  Technical  Writer,  and 

■  organizational  attributes  such  as  duration  of  work  day,  duration  of  work 
week. 

4.  Model,  Simulate,  and  Validate  the  Baseline  Processes 

In  this  step,  a  model  of  one  alternative,  (FEDOS)  was  created  from  the  baseline 
data  set  in  POW-ER,  ARENA  and  in  CORE.  (A  model  of  each  of  the  other  four 
alternatives  is  created  in  step  7.)  Figure  29  illustrates  the  data  set  inputs  to  each  modeling 
tool  and  the  outputs  from  the  simulations. 
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Baseline  Data  Set 
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■Task  Priority- 
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■IDEFO  Diagrams- 
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■Elasticity  of  Duration- 
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Figure  29.  Input  and  Output  of  Baseline  M&S. 


CORE  provided  IDEFO  Diagrams  for  functional  analysis  (see  Appendix  F).  POW-ER 
provided  Days  to  Plan  Evaluation.  Arena  provided  labor  hours  (used  for  Life  Cycle  Cost 
Estimate)  and  Elasticity  of  Labor  and  Duration  for  each  alternative. 


5.  Map  the  Alternatives  to  the  Baseline  Processes 

Each  task  in  each  of  the  alternative  tasks  was  mapped  (where  possible)  to  the 
baseline  processes.  This  allowed  the  test  team  to  use  known  labor  hours  and  cost  from  the 
baseline  to  complete  an  estimation  of  cost  and  duration  for  alternative.  Figure  30  shows 
some  of  the  tasks  from  the  baseline  and  their  mapping  to  the  JC3M  functional  hierarchy. 
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The  JC3M  Modeling  Team 
Mapped  Baseline  Processes 
to  the  JC3M  Functional 
Hierarchy  in  order  to 
associate  real  world  data  to 
the  JC3M  Functions 


1 

r 

2 

Baseline  Planning  Process  / 

\  JC3M  Functional  Hierarchy 

3 

Task  Niinie  I 

Task  Name 

1 

1.1  Elicit  Hardware  and  Software  Version  Requirements 

5 

1.1.1  Elicit  HW/SW  versions  from  the  Customer  (Create  and  Circulate  Questionaire  and  attend  weekly  WG  meetings) — 

j>  1.2.1  ID  SUT 

6 

1.1.2  Elicit  HW/SW  Delivery  Dates  (Fill  out  the  Questionaire  and  attend  weekly  WG  meetings) - 

t  1.2.1  ID  SUT 

7 

1.1.3  Create  System  s  List  (Weekly  updates  to  this  list  and  rework  as  changes  come  in) 

►  1.2.1  ID  SUT,  111  ID  Short  Falls 

8 

1.1.4  Circulate  the  System's  List  for  Customer  Review  (Attend  weekly  WG  meetings  and  send  out  a  weekly  update) - 

►  1.2.1  ID  SUT 

9 

1.1.5  Get  Customer  concurrence  for  System's  List  at  Control  Gate  1 - 

►  11.2  Conduct  Review 

V 


Figure  30.  Mapping  of  Baseline  Tasks  to  JC3M. 

This  figure  illustrates  the  traceability  of  tasks  to  the  JC3M  Functional  Flierarchy.  In  the 
figure,  processes  from  the  baseline  are  mapped  to  JC3M  tasks.  Because  many  tasks  were 
common  from  alternative  to  alternative,  baseline  tasks  could  be  mapped  to  each 
alternative  model,  ensuring  consistency  of  comparisons,  traceability  of  inputs  and 
outputs,  and  reducing  variability. 

6.  Create  Data  Sets  for  Each  JC3M  Alternative 

After  the  mapping  was  complete,  each  alternative’s  data  sets  were  assembled 
from  the  baseline  data.  Each  alternative’s  tasks  and  sub-tasks  were  mapped  as  close  as 
possible  to  baseline  data.  The  duration  and  labor  hours  were  assigned  based  on  this 
actual  data.  If  an  alternative  had  tasks  that  did  not  map  to  the  baseline,  MCTSSA  SME 
input  was  used  to  estimate  and  validate  duration  and  labor  hours. 

7.  Build  POW-ER,  ARENA,  and  CORE  Model  for  Each  Alternative 

In  step  4  a  POW-ER,  Arena,  and  CORE  model  was  developed  for  the  Baseline.  In 
step  7  a  model  of  each  of  the  remaining  four  alternatives  was  developed  in  each  of  the 
three  modeling  tools.  Figure  3 1  depicts  a  high  level  view  of  this  step. 
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Input  to  Models 


Output 


Figure  3 1 .  Input  and  Output  of  Alternative  M&S. 


This  is  a  diagram  of  Alternative  inputs  to  CORE,  POWER,  and  ARENA,  and  their 
associated  simulation  results. 


8.  Run  the  Models  and  Capture  Simulation  Results 

Each  POW-ER  model  was  run  to  generate  the  Days  to  plan  an  evaluation  EM. 
For  each  ARENA  model,  simulations  were  run  to  generate  Elasticity  of  Labor  and 
Elasticity  of  Duration  EM.  Each  CORE  model  generated  IDEFO  diagrams. 

D.  BASELINE  MODEL  DEVELOPMENT 

Each  JC3M  alternative  system  represented  test  planning  processes.  In  order  to 
compare  each  alternative’s  performance,  each  model’s  parameters  must  be  consistent  in 
terms  of  number  of  systems  under  test  and  number  of  capabilities  under  test. 

1.  Baseline  C4I  Architecture 

The  team  elected  to  model  the  C4I  SoS  that  was  under  test  during  the  FEDOS 
2005  test  event  as  the  baseline  system.  This  C4I  SoS  was  selected  because  it  had 
had  complete  employee  time  sheets,  a  project  schedule,  and  the  TSP  data  that  recorded 
the  cost  associated  with  the  planning  process.  This  data  gave  the  team  valuable  insight 
actual  labor  hours,  duration,  and  cost  to  plan  a  SoS  test  event.  The  architecture  for  the 
FEDOS  2005  event  SoS  consisted  of  ninety-one  physical  systems  comprised  of  nineteen 
types  under  test,  as  listed  in  Table  7. 
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Systems 

Software  /  Firmware  Version 

Hardware  Version 

AFATDS 

6.4.0.0UMR92Z 

CCU2/Tadpole 

CLC2S 

2.0.5 

T-40 

CONDOR 

3.6.5.0.0.9 

N/A 

DDACT 

4.0.0.0.12D 

RPDA  5500 

ENM 

4.4 

CF28 

EPLRS 

11.4 

RT-1720 

GCCS-J 

4. 0.1.0 

Netra  240 

GCCS-J  Client 

4. 0.1.0 

T-40 

IOS  VI 

4.0.1.1.01 

Netra  240 

IOS  V2  (Apps) 

3.7.0.1.01 

Netra  240 

IOS  V2  (SDS) 

3.7.0.1.01 

Netra  1 125 

IOW  VI 

4.0.1.1.07 

T-40 

IOW  V2 

3.6.6.1.01 

T-40 

MDACT 

3.6.5.0.12  Final 

RHC31A 

PFED 

1.0 

RPDA  5500 

SINCGARS 

N/A 

RT-1523A,  RT-1523B,  RT-1523E 

SISTIM 

6.4 

CCU2 

TBMCS  Lite 

1.1 

Sunfire 

TBMCS  Remote 

1.1 

T-40 

Table  7.  Systems  Under  Test  during  FEDOS  2005. 

The  architecture  for  the  FEDOS  ’05  SoS  evaluation  consisted  of  nineteen  types  of  C4I 
systems.  Ninety-one  physical  systems  were  utilized  in  the  test.  Both  system  types  and 
physical  systems  were  utilized  based  on  stakeholder  request,  doctrine,  and  equipment 
availability. 

The  systems  architecture  was  based  on  doctrine,  requests  by  stakeholders,  and 
equipment  availability  at  MCTSSA.  Figure  32  is  a  diagram  of  a  similar  C4I  SoS 
architecture,  used  for  MC3T,  in  a  2007  test  event. 
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Geographical  View  of  the  System  of 
Systems 


sscc 


]  MCTSSA 


Figure  32.  Representative  C4I  SoS  Architecture. 

This  illustration  depicts  the  echelons  of  command  seen  in  a  representative  C4I  SoS  test 
event,  as  well  as  connectivity  between  nodes.  Green  nodes  represent  notional  Marine  Air 
Wing  and  Marine  Infantry  Division  elements  at  JITC.  Yellow  nodes  represent  notional 
Marine  Expeditionary  Force,  Infantry  Division,  and  Artillery  Regiment  elements  at 
MCTSSA.  The  blue  node  represents  a  simulated  aircraft  at  Naval  Air  Warfare  Center 
(NAWC)  China  Lake.  The  white  cloud  represents  connectivity  through  the  Defense 
Research  and  Engineering  Network  to  remote  sites. 

2.  Baseline  System  of  System  Capabilities 

SoS  capabilities  are  capabilities  that  are  achievable  by  multiple  systems  working 
together  as  a  whole,  but  are  not  achievable  by  a  single  system  working  on  its  own.  The 
FEDOS  2005  test  event  included  fourteen  capabilities.  Table  8  lists  the  fourteen  SoS 
capabilities  tested. 

3.  Defining  “Old”  versus  “New”  Capabilities 

The  Arena  models  included  a  function  to  model  the  effort  required  to  plan  a  C4I 
SoS  with  both  “Old”  and  “New”  capabilities.  In  this  context,  an  “Old”  capability  is  one 
that  has  been  previously  evaluated  by  the  test  organization.  The  distinction  between 
“Old”  and  “New”  allowed  the  simulation  of  the  effects  of  learning  over  time.  The  test 
organization  familiar  with  a  capability  may  have  been  able  to  reuse  previous  or  modify 
previous  work  products.  For  a  “New”  capability,  the  test  organization  had  no  reusable 

work  products,  and  may  have  required  some  training  to  exercise  the  capability  correctly. 
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As  a  result,  new  capabilities  historically  take  longer  to  prepare  for,  regardless  of  the 


systems  involved. 


SoS  Capability 

Description 

Blue  Position 

Location  Information 
(Blue  PLI) 

This  capability  exercises  the  SoS  ability  to  receive  and  display  all 
updated  unit  positions  for  maintaining  the  Blue  Force  Common 
Operational  Picture  (COP) 

Close  Air  Support 
(CAS) 

This  capability  exercises  the  SoS  ability  to  introduce  a  Joint  Tactical  Air 
Request  (JTAR)  into  the  SoS,  approve  a  mission,  and  route  that  mission 
between  AFATDS  and  TBMCS 

Call  for  Fire  (CFF) 

This  capability  exercises  the  SoS  ability  to  transmit  a  Fire  Request  from 
a  Portable  Forward  Entry  Device  (PFED)  to  an  Artillery  battery  via 
AFATDS. 

Red  Track  Processing 

This  capability  exercises  the  SoS  ability  to  use  MAGTF  intelligence 
systems  to  process  and  fuse  enemy  location  (Red  Track)  information 

Time  Sensitive 
Targeting  (TST) 

**NEW**  This  capability  exercises  the  SoS  ability  to  use  Fires  and 
Intelligence  systems  to  process  a  Mission  Report. 

Air  Tasking  Order 
(ATO) 

This  capability  exercises  the  SoS  ability  to  generated  and  disseminated 
an  ATO  throughout  the  AFATDS  network  to  each  Effects  Management 
Tool  (IOW  VI)  using  the  MEB  TBMCS  Lite 

Battlefield  Geometry 
Exchange  (BGE) 

This  capability  exercises  the  SoS  ability  to  exchange  Battlefield 
Geometries  and  Overlays  between  the  IOS  VI  and  AFATDS 

Common  Logistics 

**NEW**  This  capability  exercises  the  SoS  ability  to  pass  logistic 
requests  between  MEB  and  RLT  CLC2S  Server  in  untethered  mode 

EPLRS  Time  Slot 
(ETS) 

This  capability  exercises  the  SoS  ability  to  compare  the  network 
utilization  of  the  EPLRS  2ms  and  4ms  time  slots.  Based  on  network 
load 

COP  Synch  Tools 
(CST) 

This  capability  exercises  the  SoS  ability  to  disseminate  tracks  within  the 
Global  Command  and  Control  System  (GCCS)  architecture,  validating 
that  the  GCCS  -  Joint  (GCCS-J),  IOS  VI,  and  GCCS-COP  (Unit 
Operations  Center)  synchronized  their  databases. 

Filters  and 

Permissions 

**NEW**  This  capability  exercises  the  SoS  ability  to  filtered  track 
information  utilizing  Geo-filters. 

Message  Exchange 
(ME) 

This  capability  exercises  the  SoS  ability  to  successfully  passed  Internet 
Relay  Chat  and  Veriable  Message  Format  (VMF)  messages  between 
company,  battalion,  and  regiment. 

Overlay  Exchange 

This  capability  exercises  the  SoS  ability  pass  VMF  K05.17  overlay  files 
message  within  the  Command  and  Control  Personal  Computer  (C2PC) 
network. 

Theatre  Ballistic 
Missile  (TBM) 

**NEW**  This  capability  exercises  the  SoS  ability  to  generate  a  missile 
track  for  dissemination  throughout  the  CST  and  C2PC  gateways. 

Table  8.  SoS  Capabilities  Tested  during  FEDOS  ’05. 

This  table  lists  the  10  old  and  4  new  SoS  capabilities  used,  at  various  command  echelons, 
in  the  FEDOS  2005  test  event.  The  capabilities  used  in  the  test  were  defined  based  on 
stakeholder  requirements,  doctrine,  and  equipment  availability. 
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E.  SIMULATION  RESULTS 

1.  Arena  Simulation  Results 

Elasticity  -  “A  measure  of  responsiveness.  The  responsiveness  of  behavior 
measured  by  variable  Z  to  a  change  in  environment  variable  Y  is  the 
change  in  Z  observed  in  response  to  a  change  in  Y.”  [About.com,  2007], 

Arena  was  used  to  calculate  elasticity  for  each  of  the  alternatives.  In  order  to 
accomplish  this,  the  team  captured  the  percent  change  in  output  ( Variable  Z)  divided  by 
percent  change  in  input  ( Variable  Y).  To  vary  the  input,  the  team  increased  or  decreased 
the  input  variables  (Number  of  Systems,  Number  of  New  Capabilities,  and  Number  of 
Old  Capabilities)  in  25%  increments  and  recorded  the  resulting  output  variables  (duration 
and  total  labor  hours).  From  the  baseline  of  10  old  and  4  new  capabilities,  and  19 
systems,  the  alternatives  were  scaled  down  in  25%  increments  to  50%  of  the  original 
scope,  and  up  in  25%  increments  to  150%  of  the  original  scope.  The  formulas  for  the 
calculation  of  the  elasticity  of  duration  and  labor  are  given  by 


Elasticity  of  Duration  = 

Z  (Defined  as  %  Change  in  Duration) 
Y  (Defined  as  %  Change  in  Input) 

(1) 

Elasticity  of  Labor  = 

Z  (Defined  as  %  Change  in  Labor) 

Y  (Defined  as  %  Change  in  Input) 

(2) 

After  an  alternative  performance  against  the  baseline  C4I  SoS  was  completed,  the 

baseline  performance  was  recorded.  The  C4I  SoS  was  scaled  down  and  up,  and  elasticity 

figures  recorded  at  each  25%  change  in  scope.  Elasticity  of  Duration  and  Elasticity  of 
Labor  were  calculated  using  the  formula  above,  and  the  results  for  each  change  in  SoS 
were  recorded.  Mean  elasticity  was  calculated  by  adding  the  four  elasticity  figures  and 
dividing  the  sum  by  four. 

Table  9  shows  the  results  of  incremental  and  mean  calculations  for  each 
alternative. 


77 


Average 

Average 

Elasticity 

Elasticity 

Elasticity 

Elasticity 

of 

of 

of 

of 

Alternative 

Duration 

Duration 

Labor 

Labor 

Baseline  (FEDOS) 

Baseline  reduced  by  50% 

0.867 

0.867 

Baseline  reduced  by  25% 

0.867 

0.867 

0.867 

0.867 

Baseline  increased  by  25% 

0.867 

0.867 

Baseline  increased  by  50% 

0.867 

0.867 

MC3T 

MC3T  reduced  by  50% 

0.782 

0.782 

MC3T  reduced  by  25% 

0.782 

0.782 

0.782 

0.782 

MC3T  increased  by  25% 

0.782 

0.782 

MC3T  increased  by  50% 

0.782 

0.782 

FCB 

FCB  reduced  by  50% 

0.717 

0.717 

FCB  reduced  by  25% 

0.717 

0.717 

0.717 

0.717 

FCB  increased  by  25% 

0.717 

0.717 

FCB  increased  by  50% 

0.717 

0.717 

SCR 

SCR  reduced  by  50% 

0.979 

0.979 

SCR  reduced  by  25% 

0.979 

0.979 

0.979 

0.979 

SCR  increased  by  25% 

0.979 

0.979 

SCR  increased  by  50% 

0.979 

0.979 

JTEM  -  CTM 

JTEM  reduced  by  50% 

0.831 

0.831 

JTEM  reduced  by  25% 

0.831 

0.831 

1.661 

1.038 

JTEM  increased  by  25% 

0.831 

0.831 

JTEM  increased  by  50% 

0.831 

0.831 

Table  9.  Elasticity  Results  for  All  Alternatives. 


Elasticity  calculations  made  from  the  output  of  the  Arena  Process  Analyzer.  Calculations 
for  the  individual  elasticity  associated  with  a  change  in  the  C4I  SoS  are  displayed  in  the 
“Elasticity  of  Duration”  column.  Mean  Elasticity  is  calculated  by  summing  the  four 
figures  for  the  alternative  and  dividing  by  four.  Within  the  limits  of  rounding,  all 
alternatives  displayed  a  consistent  elasticity  across  the  scale  changes  of  the  SoS.  The 
only  exception  was  the  JTEM  CTM  alternative,  which  showed  a  significant  elasticity 
when  decreased  to  75%  of  the  original  SoS,  and  otherwise  exhibited  the  same  elasticity 
(0.831)  at  all  other  ranges. 
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2.  Power  Simulation  Results 

POW-ER  was  used  to  generate  the  date  for  the  “Days  to  Plan”  EM.  After 
simulating  the  performance  of  each  alternative,  the  results  of  the  simulations  were 
calculated  and  expressed  as  a  number  of  8-hour  days,  using  the  organization  and 
resources  assigned  to  each  alternative.  Table  10  illustrates  the  calculated  duration  of  the 
planning  process  for  each  alternative. 


Alternative 

POW-ER  Estimation  of  Days  to  Plan  a  C4I  Evaluation 

Baseline  (FEDOS) 

140  Days 

MC3T 

121  Days 

JTEM  -  CTM 

127  Days 

FCB 

73  Days 

SCR 

158  Days 

Table  10.  POW-ER  Simulation  Results  for  JC3M  Alternatives 

This  table  provides  the  results  of  POW-ER  Simulation  and  displayed  the  estimated  total 
number  of  days  each  alternative  requires  to  plan  the  C4I  SoS  evaluation.  This  data  is  a 
critical  EM,  and  will  serve  as  input  to  the  AoA. 

3.  Results  of  Model  Validation 

The  Labor  Hours  results  from  both  the  POW-ER  and  Arena  models  were 
validated  against  the  FEDOS  2005  results.  Table  1 1  below  shows  the  Labor  Hours 
reported  from  POW-ER  and  Arena  as  well  as  from  the  FEDOS  empirical  data.  As  shown 
in  Table  1 1 ,  the  table  shows  that  the  Labor  Hours  reported  by  Arena  was  within  1 .0  % 
when  compared  to  the  historical  source  data,  and  the  hours  reported  by  POW-ER  were 
exactly  the  same  as  the  FEDOS  Labor  Hours.  The  duration  of  the  planning  phase  of  the 
evaluation,  as  reported  by  Arena,  was  at  102%  of  the  actual  duration  of  the  historical 
event,  within  the  ±4%  range  set  by  the  team.  The  duration  as  reported  by  POW-ER  was 
96.55%  of  the  actual,  and  within  the  ±4%  range  set  by  the  team.  This  provided  the  team 
with  a  high  level  of  confidence  that  the  models  accurately  represented  the  FEDOS  2005 
processes,  and  that  the  models  were  realistic  representations  of  those  processes. 
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LABOR  HOI 

URS 

RESULTS 

Alternative 

Known  Input 

Known  Output 
(Labor  Hours) 

Arena  Output 
(Labor  Hours) 

POW-ER  Output 
(Labor  Hours) 

Baseline 

(FEDOS) 

19  Systems, 

4  New  Capabilities, 
10  Old  Capabilities 

6,482 

6,484 

(Validated) 

6,482  (Validated) 

DURATION 

RESULTS 

Alternative 

Known  Input 

Known  Output 
(Duration) 

Arena  Output 
(Duration) 

POW-ER  Output 
(Duration) 

Baseline 

(FEDOS) 

19  Systems, 

4  New  Capabilities, 
10  Old  Capabilities 

145  Days 

148  Days 
(Validated) 

140  Days 
(Validated) 

Table  11.  Validation  of  Labor  Hours. 

POW-ER  and  Arena  models  were  validated  by  ensuring  the  simulation  results  were 
within  one  percent  of  the  known  data  from  historical  sources. 


F.  SUMMARY 

M&S  enabled  the  team  to  populate  the  JC3M  Scoring  Matrix  with  data  from  the 
simulation  results.  The  matrix  is  the  raw  data  that  is  needed  for  the  AoA,  which  enables 
decision-makers  to  select  the  best  alternative  from  a  utility  curve.  Table  12  illustrates  the 
results  provided  by  M&S. 


Percentage 
of  Traceable 
Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 

of 

Labor 

Elasticity 

of 

Duration 

Alternatives 

1.3.2 

1.4.3 

1.4.3 

4.1 

4.1 

FEDOS 

140  days 

0.87 

0.86 

MC3T 

121  days 

0.78 

0.78 

ITEM  CTM 

73  days 

1.04 

0.83 

FCB 

158  days 

0.97 

0.97 

SCR 

127  days 

0.71 

0.71 

Table  12.  M&S  Contribution  to  JC3M  Scoring  Matrix. 

Evaluation  Measures  used  for  comparing  alternatives  generated  by  M&S  include 
elasticity  and  duration.  Labor  hours  (not  shown)  are  used  to  calculate  LCCE. 
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V.  LIFE  CYCLE  COSTS 


A.  JC3M  LIFE  CYCLE  COST  OVERVIEW 

1.  Introduction 

LCCE  was  a  critical  component  of  the  System  Engineering  Design  Process, 
because  it  allowed  decision  makers  to  quantify  the  costs  of  each  alternative  solution  and 
compare  them  to  other  system  engineering  parameters  such  as:  percentage  of  traceable 
measures,  days  to  plan  evaluation,  quality  of  planning  outputs,  and  elasticity  of  labor  and 
duration.  Decision  makers  could  then  assess  the  risks  and  benefits  of  implementing  each 
of  the  JC3M  alternatives.  This  chapter  will  explain  the  cost  estimation  methods 
considered,  the  estimation  approach  used,  how  cost  data  was  collected,  and  provide  an 
analysis  of  the  results. 

The  LCCE  provided  an  estimate  of  the  total  life  cycle  cost  for  each  of  the  five 
alternative  JC3M  solutions  over  a  ten  year  period.  JC3M  could  be  implemented  and  used 
for  more  than  ten  years,  or  be  replaced  sooner.  The  team  expected  a  new  C4I  SoS 
evaluation  system  to  be  developed  in  order  to  provide  additional  evaluation  capability  in 
response  to  changing  conditions,  technology,  and  doctrine,  and  chose  ten  years  as  a 
representative  duration  for  JC3M. 

The  LCCE  was  limited  to  estimating  the  costs  of  planning  a  C4I  SoS  evaluation. 
The  design  of  JC3M  assumed  the  use  of  existing  processes  for  both  the  conduct  and 
reporting  of  C4I  SoS  evaluations;  test  organizations  implementing  the  JC3M  system  were 
not  expected  to  see  cost  changes  for  these  processes.  The  cost  of  buildings,  C4I  hardware 
and  software,  and  supporting  infrastructure  required  to  conduct  C4I  SoS  evaluations  were 
not  considered  in  the  LCCE.  These  items  were  considered  to  be  essential  resources  for 
any  organization  already  conducting  C4I  SoS  evaluations,  and  were  considered  sunk 
costs,  i.e.  a  cost  incurred  in  the  past  that  will  not  be  affected  by  current  or  future  decisions 
[OMB  Circular  A-94,  1992],  Sunk  costs  were  not  considered  in  the  LCCE  or  comparison 
of  alternatives  [Thuesen  and  Fabrycky,  2001:  24], 
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The  life  cycle  phases  of  the  JC3M  system,  used  in  the  LCCE,  were  Development 
and  Implementation,  Operations  and  Support,  and  Transition  and  Disposal. 

The  development  phase  included  interaction  with  stakeholders  to  begin  the 
requirements  process,  designing  the  functional  architecture,  modeling,  trade  studies,  and 
analysis  of  alternatives  [Buede,  2000].  Implementation  was  the  phase  in  which  the 
alternative  solution  was  tailored  for  use  under  local  operating  conditions.  Operations  and 
Support  (O&S)  was  the  phase  in  which  operations,  maintenance,  and  support  of  the 
JC3M  systems  and  associated  support  equipment  took  place.  Transition  and  Disposal 
was  the  phase  in  which  JC3M  would  be  retired. 

In  generating  the  LCCE,  a  mix  of  personnel,  infrastructure,  and  facilities  was 
stipulated.  The  resource  pool  identified  to  conduct  JC3M  alternatives  consisted  of  seven 
Senior  System  Analysts  who  identified  system  capabilities  and  defined  and  documented 
test  cases;  six  Senior  Tech  Specialists  who  planned,  designed,  coordinated,  and  controlled 
the  evaluation;  and  one  Senior  Program  Manager  who  managed  and  coordinated  the 
project  and  program  activity.  Physical  facilities  included  laboratory  spaces,  network 
connectivity  and  infrastructure,  and  adequate  communications.  An  organization  with  a 
significantly  different  resource  pool  or  physical  plant  will  experience  different  O&S  costs 
than  what  was  estimated  in  this  project. 

a.  Development  and  Implementation  Phase 

The  Development  and  Implementation  phase  included  the  design, 
development,  and  procurement  of  systems  and  support  items  necessary  to  conduct 
planning  of  C4I  SoS  evaluations.  In  estimating  the  cost  of  implementation  for  the 
alternatives,  published  cost  estimation  information  and  interviews  with  SME  were  used  to 
obtain  the  necessary  information.  Interviews  were  performed  with  MCTSSA  Resource 
Manager  [Manning  (c),  2007],  Architecture  Branch  Head  [Nguyen,  2007],  Data  Analysis 
Lab  Manger  [Chance,  2007],  FEDOS  SME  [Hoesly,  2007],  MC3T  IPT  Lead  [McLean 
(b),  2007]  and  Technical  Lead  [Finn,  2007],  These  interviews  along  with  Technical 
Support  Plans  [Manning  (b),  2005]  and  Consolidated  Requirement  Assessment  [McLean 
(a),  2007]  were  the  source  of  the  baseline  and  MC3T  cost  estimates.  Based  on  SME 
interviews,  historical  financial  data  from  the  2005  FEDOS  test  event,  and  monitoring  the 
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progress  of  MC3T  at  MCTSSA,  the  team  determined  that  the  development  and 
implementation  phase  of  the  JC3M  system  would  last  approximately  one  year. 

b.  Operation  and  Support  Phase 

The  Operation  and  Support  (O&S)  phase  included  operation,  maintenance, 
and  support  of  the  systems  and  associated  support  equipment  over  the  life  of  the  system. 
As  noted,  the  LCCE  only  included  the  costs  of  planning  a  C4I  SoS  evaluation.  The  O&S 
phase  was  determined  to  have  a  duration  of  nine  years  from  the  year  of  implementation. 
The  team  expected  that  by  the  end  of  the  ninth  year  the  JC3M  system  would  be  replaced 
as  legacy  systems  are  retired  and  new  systems  are  designed  and  developed  as  part  of  the 
SoS.  As  the  JCIDS  process  continues  to  develop,  more  systems  should  be  “bom  joint” 
and  their  capabilities,  as  part  of  the  C4I  SoS,  should  be  understood.  JC3M  will  need  to 
be  continually  evaluated  and  updated  to  accommodate  changes  in  technology,  system  and 
SoS  operations,  and  concepts  of  employment. 

c.  Transition  and  Retirement  Phase 

The  Transition  and  Retirement  phase  included  costs  associated  with  the 
termination  and  retirement  of  the  process.  If  the  O&S  phase  was  extended  beyond,  or 
reduced  from,  nine  years  this  phase  would  take  place  later  or  earlier,  but  was  still 
projected  to  last  approximately  one  year.  The  team  estimated  that  the  cost  of  this  phase 
would  be  minimal  due  to  the  conceptual  nature  of  the  activities  conducted  by  the  JC3M 
system.  The  three  phases  of  the  LCCE  are  graphically  displayed  in  Figure  33. 
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Figure  33.  Phases  of  JC3M  Life  Cycle. 

The  three  life  cycle  phases  of  the  JC3M  alternatives  under  consideration  are 
Development  and  Implementation,  Operations  and  Support,  and  Transition  and 
Retirement.  This  graphically  displays  the  phases  over  the  ten  year  life  cycle. 
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2. 


Life  Cycle  Cost  Elements 

a.  Development  and  Implementation  Costs 
The  development  and  implementation  cost  for  JC3M  was  treated  as  a 

non-recurring  cost.  The  targeted  customers  of  the  JC3M  system  were  assumed  to  be  DoD 
organizations  involved  with  C4I  SoS  test  and  evaluation.  The  cost  for  developing  and 
implementing  JC3M  came  primarily  from  process  documentation  and  initial  training. 
Because  of  the  conceptual  nature  of  JC3M,  there  were  no  costs  expected  from  the 
procurement  of  software  packages,  software  licenses,  or  hardware  additions  or  upgrades. 

b.  Operation  and  Support  Costs 

O&S  cost  is  the  major  portion  of  the  life  cycle  cost  of  JC3M.  The  O&S 
Cost-Estimating  Guide  [Cost  Analysis  Improvement  Group,  2002]  defines  O&S  cost  as 
the  “costs  of  personnel,. . .  services,  and  sustaining  support. . ..”  associated  with  any 
system.  To  arrive  at  a  realistic  estimate  of  JC3M  over  its  life  cycle,  current  information 
was  gathered  from  a  variety  of  interviews  (see  Section  A.l.a  “Development  and 
Implementation”)  and  financial  data  from  the  FEDOS  TSP  [Manning  (b),  2005]  and 
MC3T  CRA  [McLean  (a),  2007],  internal  MCTSSA  documents  used  to  forecast  and 
manage  the  respective  projects.  The  O&S  cost  elements  included  the  labor  cost  of 
personnel  required  to  plan  a  C4I  SoS  evaluation;  follow  on  training;  and  the  software  and 
hardware  periodic  upgrades,  license  renewals,  and  maintenance  contracts  required  for 
planning  C4I  SoS  evaluation. 

c.  Transition  and  Retirement  Costs 

The  transition  and  disposal  cost  of  the  JC3M  system  consisted  of  the 
manpower  costs  of  personnel  required  to  provide  customer  support  during  the  transition 
period,  document  archiving,  and  configuration  management.  Customer  support  was 
estimated  to  require  the  services  of  at  least  one  engineer  to  work  with  users  to  facilitate 
the  teardown  and  disposal  of  systems  and  at  least  one  configuration  management 
specialist  to  assist  in  archiving  relevant  documents  and  software.  This  phase  was 
estimated  to  last  no  more  than  one  year  after  the  end  of  the  O&S  phase  of  the  JC3M  life 
cycle.  The  majority  of  JC3M  documents  would  be  electronically  stored  and  managed; 
therefore,  costs  in  this  phase  are  associated  with  computers  and  online  storage  space.  The 
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cost  to  transition  and  retire  the  planning  processes  associated  with  JC3M  are  considered 
to  be  minimal  due  to  the  conceptual  nature  of  JC3M. 

3.  Cost  Estimation  Assumptions  and  Constraints 

The  cost  estimate  assumptions  and  constraints  are  summarized  as  follows: 

■  The  application  of  JC3M  processes  did  not  require  any  additional 
equipment  or  tools. 

■  JC3M  planning  processes  are  conceptual  and  no  specialized  software  or 
hardware  was  required. 

■  Specialized  education  or  training  was  not  required  in  applying  the  JC3M 
process. 

■  The  cost  for  developing  and  implementing  JC3M  was  a  one-time  non¬ 
recurring  cost. 

■  The  cost  for  the  disposal  phase  was  minimal  due  to  the  conceptual  nature 
of  JC3M. 

■  Overhead  cost  in  each  organization  was  not  considered. 

■  The  team  recognized  the  variance  in  duration  of  “Days  to  Plan”  among  the 
alternatives,  but  for  consistency,  elected  to  consider  each  alternative 
conducting  one  C4I  SoS  evaluation  per  year. 

B.  COST  ESTIMATION  METHODS  CONSIDERED 

The  JC3M  LCCE  was  completed  using  a  combination  of  the  Analogy  cost  method 
[Defense  Acquisition  Guidebook,  2006]  and  the  Activities-Based  Costing  (ABC)  method 
[Blanchard  and  Fabrycky,  1998:  580].  These  methods  are  described  below. 

Estimation  by  Analogy  is  employed  when  the  project  under  consideration  is 
similar  to  a  previously  fielded  or  implemented  system.  Estimation  by  analogy  requires 
that  current  systems  similar  in  design  and  function  are  selected  for  the  basis  of  cost 
estimation  [Defense  Acquisition  Guidebook,  2006], 

MC3T  is  a  system  designed  for  defining,  documenting,  and  evaluating  the 
performance  of  a  MAGTF  C4I  SoS  as  a  single  system,  in  accordance  with  modem 
systems  engineering  practices.  Because  JC3M  was  designed  to  perform  functions  similar 
to  those  of  MC3T,  the  team  determined  that  analogy  was  an  appropriate  method  of  cost 
estimation.  Historical  cost  data  from  the  MC3T  system  was  used  to  estimate  the  cost  of 
development  and  implementation  of  MC3T.  MC3T  cost  data  was  obtained  primarily 
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from  interviews  with  the  MC3T  IPT  Lead  McLean  (a)  [2007],  and  the  draft  CRA 
[McLean  (b),  2007], 

ABC  is  used  when  estimating  the  cost  to  carry  out  the  activities  and  processes  of  a 
system.  The  main  goal  of  this  methodology  is  to  ensure  that  all  costs  are  traceable  back 
to  the  functions  or  processes  that  actually  occurred  within  the  system.  In  the  ABC 
methodology,  costs  are  directly  traceable  to  the  applicable  cost-generating  process  and 
product.  Cost  can  be  easily  estimated  on  a  functional  basis,  and  the  emphasis  is  on 
resource  consumption.  Processes  and  products  are  performed  by  activities,  and  activities 
consume  resources.  The  ABC  approach  fosters  the  establishment  of  cause-and-effect 
relationships,  enables  the  identification  of  the  high-cost  contributors,  and  eliminates 
double  counting  that  is  typically  inherent  with  the  application  of  direct  and  indirect  costs. 
[Blanchard  and  Fabrycky,  1998:  580-581],  The  ABC  method  was  used  to  estimate  O&S 
cost  for  the  JC3M  alternatives. 

The  JC3M  system  utilized  resources  to  execute  functions  and  within  these 
functions  processes  and  tasks  were  executed  in  order  to  produce  the  desired  outputs.  The 
cost  of  each  process  or  task,  in  labor-hours  consumed,  was  directly  traceable  back  to  the 
cost-generating  function.  The  input  to  the  cost  model  was  based  on  the  labor  hours 
required  to  execute  the  functions  associated  with  each  of  the  alternatives. 

The  team  also  considered  the  Engineering  Estimate  method  and  the  Actual  Cost 
method  [Defense  Acquisition  Guidebook,  2006],  The  engineering  estimation  method  is 
employed  when  the  system  is  broken  down  into  components.  However,  since  the  JC3M 
system  consists  primarily  of  activities,  this  methodology  was  not  suitable.  The  Actual 
Cost  Estimation  method  was  considered  but  because  there  was  no  actual  cost  experience 
or  trends  from  prototypes,  engineering  development  models,  or  early  production  items  to 
project  estimates  of  future  costs  this  method  was  also  rejected. 
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C.  JC3M  COST  ESTIMATION  APPROACH 

The  team  followed  the  cost  estimation  approach  described  by  Blanchard  and 
Fabrycky  [1998:  567]  to  estimate  the  costs  for  the  JC3M  system.  The  approach  was 
tailored  to  fit  the  requirements  of  the  JC3M  system  as  defined  in  Chapter  II  “Problem 
Refinement.” 

The  cost  model  was  designed  to  estimate  the  life  cycle  cost  of  planning  C4I  SoS 
evaluations  as  conducted  by  each  of  the  alternative  solutions.  The  cost  model  was 
constructed  using  Microsoft  Excel®  spreadsheet  software.  The  cost  model  calculated  the 
cost  of  each  task,  year,  life  cycle  phase,  and  overall  cost  for  each  alternative,  and  the 
costs  for  each  alternative  were  summarized. 

In  each  phase  of  the  JC3M  life  cycle,  tasks  and  sub-tasks  were  executed  in  order 
to  support  the  functions  of  the  system.  These  tasks  and  sub-tasks  were  identified  based  on 
the  functional  decomposition  of  the  system.  The  output  values  from  the  LCCE  provided 
the  cost  inputs  to  the  AoA. 

The  JC3M  Cost  Breakdown  Structure  (CBS)  used  a  hierarchical  framework  for 
estimating  the  life  cycle  cost.  The  CBS  provided  a  structure  that  identified  the  elements 
or  categories  of  costs  that  were  incurred  in  each  phase  of  the  JC3M  life  cycle.  It  also 
served  as  a  guide  for  cost  data  collection  for  each  of  the  high  level  functions  of  the  JC3M 
system.  Because  each  of  the  alternatives  was  unique,  not  all  the  alternatives  had  cost  data 
for  the  identified  cost  categories.  The  FEDOS  alternative  did  not  define  performance 
measures,  and  did  not  report  a  cost  for  this  task. 

The  CBS  defined  the  specific  elements  included  in  the  cost  estimate  and 
organized  the  LCCE  results  in  a  hierarchy  [Defense  Acquisition  Guidebook,  2006],  The 
CBS  identified  the  cost  elements  for  each  phase  and  guided  cost  data  collection  for  each 
of  the  high  level  functions  of  JC3M.  Figure  34  is  a  graphical  representation  of  the  JC3M 
CBS. 
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Figure  34.  Cost  Breakdown  Structure  of  JC3M. 

The  CBS  provided  a  structure  that  identified  the  categories  of  costs  in  each  phase  of  the 
JC3M  life  cycle.  It  also  served  as  a  guide  line  for  cost  data  collection  for  each  of  the  high 
level  functions  of  the  JC3M  system. 

D.  COST  DATA  COLLECTION 

Data  for  this  effort  was  extracted  from  the  MCTSSA  CRA  [McLean  (a),  2007] 
that  provided  resource  requirement  assessments  for  funding,  staffing,  data  collection  and 
monitoring  networks,  data  capture,  and  information  assurance.  The  O&S  cost  of  JC3M 
was  most  influenced  by  the  direct  labor  costs  of  the  technical  staff. 


The  MCTSSA  test  team  consisted  of  federal  civilian  employees,  contractors,  and 
a  small  military  staff  that  provided  C4I  systems  support,  operation,  and  management. 
Fully  burdened  labor  rates  for  civilians  and  contractors  were  provided  by  the  MCTSSA 

comptroller  office  for  Fiscal  Year  (FY)  2007  [George  (a),  2007].  Fully  burdened  FY 
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2007  labor  rates  [Military  Composite  Pay  and  Reimbursements  Rate  Table,  2007]  were 
used  to  calculate  cost  of  military  personnel. 

The  MC3T  IPT  lead  [McLean(b),  2007]  was  interviewed  to  obtain  information  on 
the  duration  of  the  development,  implementation,  and  operations  of  MC3T.  Projected 
personnel  labor  hours,  software  and  hardware  cost,  as  well  as  the  number  of  billets 
required  were  obtained  from  the  draft  version  of  the  CRA  provided  by  the  MC3T  IPT 
Lead  [McLean  (a),  2007],  The  CRA  was  the  primary  source  of  data  for  the  cost 
estimation.  The  team  also  interviewed  stakeholders  and  subject  matters  experts  to  obtain 
cost  data  and  information  that  was  not  readily  available.  In  calculating  the  personnel 
labor  costs,  the  fully  burdened  rate  for  government  civilians  [DoD  Civilian  Personnel 
Management  Service,  2007],  military  [Office  of  the  Under  Secretary  of  Defense  for 
Personnel  and  Readiness,  2007],  and  contractor  billets  [GSA  schedule  for  Systems 
Engineering  Support  Service]  were  used.  Interviews  with  the  MCTSSA  assistant 
comptroller  [George  (b),  2007]  were  also  conducted  to  provide  additional  information  on 
the  fully  burdened  labor  rates  for  civilians  and  contractors  at  MCTSSA. 

E.  ANALYSIS  OF  COST  DATA 

1.  Development  Costs 

Development  costs  for  the  FEDOS  alternative  were  recorded  as  $0  in  the  LCCE, 
because  this  alternative  was  fully  developed.  Development  costs  for  the  MC3T 
alternative  were  recorded  as  $0  in  the  LCCE,  because  this  alternative  was  considered 
fully  developed.  Resources  used  for  development  of  the  MC3T  alternative  were 
identified,  based  on  review  of  historical  data  and  SME  interviews,  and  used  by  analogy  to 
estimate  the  development  costs  of  both  the  FCB  and  SCR  alternatives  which  did  not  have 
historical  development  costs.  FCB  and  SCR  estimated  developments  costs  are  adjusted 
to  account  for  specific  differences  between  the  alternatives.  Development  costs  for  the 
ITEM  CTM  alternative  were  based  on  SME  interviews,  and  are  described  below.  All 
development  costs  are  recorded  in  the  LCCE  and  summarized  in  Table  13. 

Development  of  MC3T  included  a  Self  Assessment  Team  (SAT),  a 
multidisciplinary  IPT  formed  to  evaluate  MC3T  processes  and  assist  in  refining  processes 
prior  to  execution.  The  SAT  also  served  to  capture  lessons  learned  and  document 
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changes  for  process  improvement.  The  estimation  of  the  labor  cost  for  the  SAT  was 
based  on  the  MC3T  POAM-OOl-Vl  [McLean  (c),  2007],  the  Requirements  for  Members 
of  the  MCTSSA  Self  Assessment  Team,  [Villar  (a),  2007]  and  an  interview  with  the  SAT 
lead  [Villar  (b),  2007], 

Colonel  Eileen  Bjorkman,  United  States  Air  Force  (USAF),  Test  Director,  Joint 
Test  Evaluation  Methodology,  stated  the  JTEM  CTM  will  not  be  fully  developed  until 
FY09,  and  estimated  the  total  remaining  cost  to  develop  the  CTM  through  2009  at  $3.5M 
[Bjorkman,  (b)  2007],  This  estimate  includes  JTEM  CTM  Community  of  Interest  work, 
documentation  and  process  writing,  and  participation  in  events.  The  JTEM  CTM 
development  cost  does  not  consider  overhead  cost  such  as  utilities,  facilities  operation 
and  maintenance,  administration  costs,  and  other  overhead  cost  not  directly  related  to 
labor  cost.  This  estimation  is  consistent  with  the  cost  estimations  used  for  development  of 
the  other  alternatives.  This  development  cost  is  allocated  in  the  cost  model  as  $1.03M  for 
the  remainder  of  year  1  (FY07),  and  $2.47M  for  year  2  (FY08). 

2.  Implementation  Costs 

Implementation  costs  for  the  FEDOS  alternative  were  not  found  in  historical  data. 
Implementation  costs  for  the  FCB,  SCR,  and  JTEM  CTM  alternatives  were  not  available 
because  they  have  not  been  used.  The  cost  to  implement  these  four  alternatives  was 
estimated  by  analogy  with  the  MC3T  implementation  costs,  because  historical  data  is 
available,  the  alternatives  perform  similar  tasks,  and  are  of  similar  complexity.  All 
implementation  costs  are  recorded  in  the  LCCE  and  summarized  in  Table  13. 

The  CRA  [McLean  (a),  2007]  provided  the  most  accurate  source  of  data  on 
personnel,  data  capture,  hardware,  and  software  costs  of  MC3T  implementation.  This 
estimate  includes  costs  of  training  personnel,  documentation,  capturing  lessons  learned, 
documenting  changes,  and  software  licensing  fees,  and  totals  approximately  $1.1M.  The 
implementation  effort  will  take  MC3T  to  the  start  of  the  first  evaluation;  the  conduct  of 
the  initial  C4I  SoS  evaluation  is  not  included. 

Implementation  of  the  JTEM  CTM  does  not  begin  until  year  3  of  the  LCCE,  when 
development  of  the  alternative  is  completed. 
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3.  Results  of  Cost  Data  Analysis 

The  JC3M  team  determined  the  most  expensive  alternative  was  the  FCB 
alternative,  at  a  cost  of  $8.13M  over  the  ten-year  projected  lifecycle.  The  team  also  noted 
that  FCB  is  approximately  $0.41M  more  than  SCR.  The  team  calculated  the  cost  of  FCB 
as  a  cost  to  DoD,  i.e.  while  the  senior  SMEs  who  generate  the  performance  measures  do 
not  charge  their  efforts  directly  to  a  C4I  test  organizations,  their  time  and  effort  is  a  cost 
to  DoD.  In  order  of  increasing  cost,  the  other  alternatives  were  FEDOS  at  $5.01M; 

MC3T  at  $5.97M;  JTEM  CTM  at  S6.97M;  and  SCR  at  S7.72M. 

The  team  determined  that  MC3T  was  estimated  to  cost  approximately  $0.96M 
more  than  FEDOS,  which  it  has  replaced.  While  this  is  a  significant  cost  difference,  the 
increase  can  be  directly  attributed  to  the  increased  in  scope,  duration,  and  level  of  effort 
involved  in  MC3T,  which,  in  addition  to  the  factual  data  cited  above,  anecdotally 
supported  the  increased  cost  of  MC3T. 

The  team  noted  the  alternative  with  the  lowest  estimated  O&S  cost  was  the  JTEM 
CTM  alternative,  at  $2.3M,  followed  by  FEDOS  at  $3.9M,  MC3T  at  $4.8M,  SCR  at 
$5.6M,  and  FCB  at  $5.8M.  This  is  significant  because  O&S  represents  the  largest 
portion  of  the  LCCE,  and  is  the  portion  directly  assumed  by  a  test  organization  that 
implements  any  alternative. 

Implementation  of  JC3M  would  modify  topics  assessed  in  a  C4I  SoS  evaluation, 
which  could  cause  cost  changes  in  the  conduct  and  reporting  phases  of  the  evaluation.  In 
part  to  assess  these  potential  changes,  the  team  recommended  validation  of  JC3M  results 
(see  Chapter  7,  Section  D  “Conclusion”)  through  a  “real  world”  evaluation. 

The  team  noted  that  due  to  the  continuing  development  of  the  JTEM  CTM 
alternative,  the  O&S  phase  would  not  start  until  one  year  after  the  other  four  alternatives. 
As  a  result,  over  the  ten  year  duration  of  the  LCCE,  the  JTEM  CTM  would  complete  one 
less  C4I  SoS  evaluation.  Table  13  summarizes  the  LCC  results. 
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Life-Cycle  Year 

Total  Cost 

Alternatives 

1 

2 

3 

4.. .9 

10 

($) 

FEDOS 

Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,052,527 

0 

0 

0 

0 

1,052,527 

Operational  &  Support. 

0 

419,497 

419,497 

419,497 

2,200 

3,908,178 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,052,527 

419,497 

419,497 

419,497 

52,200 

5,010,706 

MC3T 

Development 

0 

0 

0 

0 

0 

0 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational  &  Support. 

0 

525,537 

525,537 

525,537 

2,200 

4,756,500 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

1,169,414 

525,537 

525,537 

525,537 

52,200 

5,975,913 

JTEM  CTM 

Development 

1,030,000 

2,470,000 

0 

0 

0 

3,500,000 

Implementation 

0 

0 

1,169,414 

0 

0 

1,169,414 

Operational  &  Support. 

0 

0 

0 

558,535 

2,253,410 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

Total  Cost 

1,030,000 

2,470,000 

1,169,414 

558,535 

52,200 

6,972,824 

FCB 

Development 

1,021,835 

0 

0 

0 

0 

1,021,835 

Implementation 

1,301,282 

0 

0 

0 

0 

1,301,282 

Operational  &  Support 

0 

650,223 

650,223 

650,223 

2,200 

5,753,985 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

2,323,117 

650,223 

650,223 

650,223 

52,200 

8,127,101 

SCR 

Development 

952,007 

0 

0 

0 

0 

952,007 

Implementation 

1,169,414 

0 

0 

0 

0 

1,169,414 

Operational  &  Support. 

0 

624,451 

624,451 

624,451 

2,200 

5,547,811 

Transition  and  Disposal 

0 

0 

0 

0 

50,000 

50,000 

Total  Cost 

2,121,421 

624,451 

624,451 

624,451 

52,200 

7,719,232 

Table  13.  Life  Cycle  Cost  Summary. 


Summary  sheet  of  life  cycle  costs  for  all  alternatives  based  on  CBS  over  a  10-year  life 
cycle 
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VI.  ANALYSIS  OF  ALTERNATIVES 


A.  METHODOLOGY 

The  team  conducted  an  Analysis  of  Alternatives  in  order  to  compare  the 
performance  of  each  of  the  five  alternatives  described  in  Chapter  III,  Design  Alternatives. 
AoA  compared  the  outputs  of  EMs  and  LCCE  for  each  alternative.  The  team  utilized  an 
approach  that  provided  a  comparison  of  total  cost  to  total  performance. 

An  Analysis  of  Alternatives  (AoA)  compares  alternatives  by  estimating 
their  ability  to  satisfy  the  identified  requirements  through  an  effectiveness 
analysis  and  by  estimating  their  life  cycle  costs  (LCC)  through  cost 
analysis.  The  results  of  these  two  analyses  are  used  together  to  produce  a 
cost-effectiveness  comparison  that  allows  decision  makers  to  [evaluate] 
cost  and  effectiveness  simultaneously  [Feuchter,  2004:  7], 

B.  MULTI-ATTRIBUTE  UTILITY  THEORY 

Multi-attribute  Utility  Theory  (MAUT)  is  a  “quantitative  method  for  aggregating 
a  stakeholder’s  preferences  over  conflicting  objectives  to  find  the  alternative  with  the 
highest  value  when  all  objectives  are  considered”  [Buede,  2000:  361],  The  five 
alternatives  under  consideration  were  evaluated  and  compared  against  each  other  in  this 
section  using  MAUT.  The  goal  of  the  analysis  was  to  provide  a  structured  method  in  the 
decision-making  process.  There  are  several  techniques  that  can  be  used  to  compare 
alternatives  when  there  are  multiple  evaluation  measures  with  different  units  of  measure. 
The  team  decided  to  use  the  Value  Modeling  technique  as  part  of  the  JC3M  SEDP. 

Sage  and  Rouse  [1999:  1 128]  stated  that  to  select  the  preferred  alternative  it  is 
necessary  to  combine  evaluation  measures  into  a  single  measure  of  the  overall 
desirability  of  an  alternative.  This  is  done  by  developing  a  value  function 
( v(Xj  ,x2, ...  ,xn)  ).  The  equation  for  the  value  function,  under  the  condition  of  linear 
additive  independence  of  attributes,  is  given  by 

v(v  ,X2,...,Xn)  =  WjVj  (Xj  )  +  w2v2  (x2  )  +  •••  +  wnvn  (xn )  ^ 

where  xj,  xi,  ...  x„  are  the  EMs  for  an  objective/requirement,  vfxi),  v^fxi),  ...  vn(x„)  are 
the  utility  scores  of  the  EMs  for  a  specific  alternative,  and  wj,  W2,  ...  w„  are  the  weights 
for  each  EM. 
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C.  VALUE  MODELING  TECHNIQUE 

Value  modeling  as  applied  to  this  project  was  discussed  in  detail  in  Chapter  II, 
Problem  Refinement.  Value  modeling  has  three  steps,  1)  the  definition  of  a  Qualitative 
Value  Model,  2)  the  definition  of  a  Quantitative  Value  Model,  and  3)  the  definition  of  an 
Additive  Value  Model.  Figure  35  provides  an  overview  of  the  structure  of  value 
modeling. 


score  of  each  alternative 


Figure  35.  High  Level  Outline  of  Value  Modeling. 

Value  Modeling  is  composed  of  Qualitative  Value  Modeling,  Quantitative  Value 
Modeling,  and  Additive  Value  Modeling.  Details  of  the  JC3M  Qualitative  Value 
Modeling  process  are  provided  in  the  Chapter  II  “Problem  Refinement.”  Quantitative 
Value  Modeling  includes  the  definition  of  relative  importance  of  EM;  Additive  Value 
modeling  converts  raw  scores  to  total  value  of  each  alternative. 

The  Critical  EMs  used  to  compare  performance  of  each  alternative  were  described 
in  Table  1,  Chapter  II,  Problem  Refinement.  The  EMs  used  in  the  functional  analysis 
were  the  same  EMs  used  to  evaluate  the  alternatives.  This  provided  clear  traceability 
between  the  functional  requirements  and  the  AoA.  In  order  to  perform  an  effective 
evaluation  of  performance  it  was  necessary  to  have  a  consistent  and  accurate  set  of 
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criteria.  These  criteria  must  be  based  on  the  design  objectives,  i.e.,  what  it  is  that  the 
design  is  meant  to  achieve  [Cross,  2000:  140].  The  criteria  used  to  evaluate  EMs  for  the 
alternatives  are  shown  in  Table  14.  Table  14  expands  on  Table  1  by  adding  the  ideal 
values  of  the  EMs. 


Percentage  of 
Traceable 
Measures 

Days  to  Plan 
Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity  of 
Labor 

Elasticity  of 
Duration 

JC3M 

Function 

Define 

Measures 

1.3.2 

Planning 

Results 

1.4.3 

Planning 

Results 

1.4.3 

Input  System 
Flexibility 

4.1 

Input  System 
Flexibility 

4.1 

Definition 

Alternative 
generated 
measures, 
traceable  to 
stakeholder 
requirements, 
divided  by  the 
number  of 

measures 
generated  by 
the  alternative. 

Sum  of  labor 
hours  required, 
expressed  in 
days,  to  plan 

C4I  SoS 
evaluation 

Assign  an 
overall  quality 
level  to  the 
planning 
documents 
produced. 

Divide  percent 
change  in  labor 
hours  to 
conduct 
planning  phase 
of  JC3Mby  the 
percent  change 
in  systems 
under  test. 

Divide  percent 
change  in 
duration  to 
conduct 
planning  phase 
of  JC3M  by  the 
percent  change 
in  systems 
under  test. 

Range 

0-100% 

>0 

Likert  Scale  of 

1  -4 

0  -  oo 

0  -  oo 

Ideal  Value 

100% 

Less  is  better 

4  is  best 

Less  is  better 

Less  is  better 

Table  14.  JC3M  Critical  Evaluation  Measures. 

Critical  EMs  used  to  compare  the  performance  of  alternative  solutions.  This  table 
expands  on  Table  1  by  including  ideal  values. 

D.  QUANTITATIVE  VALUE  MODEL 

The  Quantitative  Value  Model  utilized  the  decision  maker’s  preferences  with 
respect  to  EMs.  This  involved  defining  utility  curves  and  weights  for  each  EM.  Utility 
curves  were  used  to  normalize  EM  scores,  i.e.,  convert  a  raw  score  received  from  M&S 
or  from  an  SME,  into  a  utility  score.  For  example,  consider  the  top  speed  of  a  vehicle  as 
an  EM  of  interest  to  a  decision  maker.  If  a  vehicle  being  considered  for  purchase  had  a 
top  speed  of  60  mph,  this  raw  value  would  be  converted  to  a  value  on  a  scale  of  0  to  1, 
where  0  represents  no  utility  to  the  stakeholder,  and  1  represents  the  highest  utility  to  the 
stakeholder.  The  “worth”  of  the  top  speed  of  60  mph  would  be  dependent  on  the 
decision  maker’s  preferences  and  could  be  modeled  with  a  utility  function.  The  utility 

score  of  EMs  were  expressed  on  a  common  scale  in  order  to  allow  useful  comparisons. 
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There  may  be  multiple  EMs  and  each  may  have  different  relative  worth  to  a 
decision  maker.  In  the  example  above,  purchase  price  may  be  more  important  to  a 
decision  maker  than  top  speed,  braking  distance,  or  acceleration.  Based  on  the 
preferences  of  the  decision  maker(s),  appropriate  weights  were  assigned  to  each  EM  in 
order  to  reflect  their  relative  importance. 

1.  Eliciting  Utility  Curves  from  Stakeholders 

The  team  evaluated  two  methods  to  implement  utility  functions,  also  referred  to 
as  Standard  Scoring  Functions  (SSFs).  [Wymore,  1993]  developed  a  set  of  twelve  SSFs. 
In  addition,  [Buede,  2000:365]  described  a  family  of  exponential  utility  functions. 
[Daniels,  2001:209]  compared  both  methods  and  concluded  that  only  eight  utility 
functions  are  required  for  all  applications  that  he  and  his  team  had  encountered.  Four  of 
the  eight  are  Wymorian  SSFs.  The  team  selected  two  Wymorian  functions:  SSF  3  and 
SSF  9,  because  they  fit  the  parameters  that  can  be  assignable  to  the  evaluation  measures 
being  modeled  by  the  team  [Daniels,  2001:206], 

The  SSF3  guideline  is:  “If  more  is  better,  the  customer  can  provide  both  a  finite 
upper  bound  and  a  finite  lower  bound,  then  choose  SSF3.  This  is  by  far  the  most  common 
scoring  function  used  when  dealing  with  an  EM  where  more  is  better”  [Daniels,  2001: 
204], 

The  SSF9  guideline  is:  “If  more  is  worse,  and  the  stakeholder  can  provide  both  a 
finite  upper  bound  and  a  finite  lower  bound,  then  choose  SSF9.  This  scoring  function  is 
by  far  the  most  frequently  used  with  EMs  where  more  is  worse”  [Daniels,  2001:  207]. 

The  five  parameters  required  to  create  Wymorian  SSFs  are  defined  as  follows: 

L  -  The  lower  threshold  of  performance  for  the  EM  below  which  the  value  to  the 
customer  is  undesirable  (but  not  necessarily  unacceptable)  and  is  assigned  a  zero  score. 

B  -  The  parameter  B  is  called  the  baseline  value  for  the  EM  and  can  be  chosen  as 
the  design  goal  or  the  status  quo  for  this  or  similar  systems.  By  definition,  baseline  values 
are  always  assigned  a  score  of  0.5. 

U  -  The  upper  threshold  of  performance  for  the  EM  above  which  the  value  to  the 
customer  is  assigned  a  score  of  one. 
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S  -  The  parameter  S  determines  the  behavior  of  the  scoring  function  in  the 
neighborhood  of  the  baseline  value  B.  Mathematically,  S  is  the  slope  of  the  tangent  to  the 
scoring  function  at  the  baseline  value  B.  The  slope  represents  the  maximum  incremental 
change  in  the  customer’s  quantitative  judgment  with  each  incremental  change  in  input. 

D  -  The  parameter  D  represents  the  domain  of  definition  of  the  scoring  function. 
D  is  defined  as  the  range  between  L  and  U.  Conceptually,  D  clearly  states  the  range  of 
input  values  that  are  possible  from  a  build-ability  viewpoint  or  legal  due  to  mandatory 
requirements.  Values  outside  this  range  constitute  impossible  or  unacceptable  inputs 
[Daniels,  2001:  203]. 


Each  SSF  figure  included  the  SSF  Value  or  Utility  Score  along  the  y  axis  and  the 
raw  EM  score  along  the  x  axis.  In  addition,  each  figure  included  the  parameters  that  can 
be  used  to  recreate  the  SSF.  Figure  36  illustrates  Wymore’s  SSF  3  and  SSF  9. 


Figure  36.  Wymorian  Standard  Scoring  Functions  3  and  9. 

SSF  3  and  SSF  9  were  chosen  by  the  team  to  use  during  AoA.  Five  parameters  were 
needed  to  create  a  Wymorian  SSF.  These  are  Lower  Limit  (L),  Baseline  Value  (B), 

Upper  Limit  (U),  Slope  (S),  and  Domain  (D).  SSF  3  was  used  to  score  measures  that  are 
more  valuable  as  they  increase  (Percentage  of  Traceable  Measures).  SSF  9  was  used  to 
score  measures  that  are  more  valuable  as  they  decrease  (Days  to  Plan  Evaluation). 

Thomas  Fee  Rodgers  in  cooperation  with  A.  Terry  Bahill  from  the  University  of 

Arizona  developed  a  software  program  that  implements  the  SSFs  described  here 

[Rodgers  and  Bahill,  2007].  The  team  used  this  software  to  implement  the  SSFs 

described  in  Figures  38  through  41. 
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The  team  constructed  initial  SSFs  for  the  five  EMs  and  presented  them  to  a  group 
of  SMEs,  as  described  in  Appendix  C.  Representatives  from  JTEM,  JITC,  NPS, 
MCTSSA,  and  NAWC  China  Lake  were  invited  to  provide  stakeholder  preferences  for 
the  Quantitative  Value  Model.  Two  representatives  from  MCTSSA,  with  United  States 
Marine  Corps  (USMC)  C4I  SoS  test  experience,  and  two  representatives  from  NAWC 
China  Lake,  with  United  States  Navy  (USN)  C4I  SoS  test  experience,  were  available  and 
consulted  to  provide  stakeholder  preferences.  Stakeholders  approved  the  SSFs  and 
provided  weights  for  each  EM  using  the  Analytical  Hierarchy  Process  (AHP). 

2.  Percent  Traceable  Measures  Utility  Curve 

Percentage  of  Traceable  Measures  was  an  EM  that  reported  on  how  well  the 
alternative  creates  performance  measures  traceable  to  authoritative  sources.  As  the 
percentage  increased,  the  performance  measures  were  more  valuable  to  the  organization 
conducting  the  C4I  SoS  evaluation.  The  SSF  used  to  score  this  EM  is  illustrated  in 
Figure  37. 


Percentage  of  Traceable  Measures 


Figure  37.  Utility  Function  for  Percentage  of  Traceable  Measures. 

This  is  the  stakeholder  validated  Utility  Function  used  for  this  EM.  A  higher  percentage 
was  more  desirable.  Values  less  than  65%  received  a  utility  of  0.  Values  greater  than  or 
equal  to  95%  received  a  utility  of  1 .0.  SSF  3  Parameters  were  L  =  65%,  B  =  80%,  U  = 
95%,  D  (U  -  L)  =  30%,  and  S  =  0.67. 
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3.  Days  to  Plan  Utility  Curve 

The  Days  to  PlanEM  reported  how  many  work  days  were  required  by  the 
alternative  to  plan  the  evaluation  of  the  baseline  C4I  SoS.  As  the  number  of  days 
decreased,  the  alternative  became  more  valuable  to  the  organization  conducting  the  C4I 
SoS  evaluation,  because  more  evaluations  could  be  planned  over  the  same  time  period,  or 
plans  could  be  performed  more  quickly.  The  SSF  used  to  score  this  EM  is  illustrated  in 
Figure  38. 
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Figure  38.  Utility  Function  for  Days  to  Plan  Evaluation. 

This  is  the  stakeholder  validated  Utility  Function  used  for  this  EM.  A  lower  number  of 
days  was  more  desirable.  Values  less  than  90  days  received  a  utility  of  1.0.  Values 
greater  than  or  equal  to  210  days  received  a  utility  of  0.  SSF  9  Parameters  were  L  =  90 
days,  B  =  150  days,  U  =  210  days,  D  (U  -  L)  =  120,  and  S  =  -  0.0167. 

4.  Quality  of  Planning  Outputs  Utility  Curve 

Quality  of  Planning  Outputs  was  an  EM  that  reported  on  overall  “goodness”  of 
the  test  plans  and  test  procedures  produced  by  the  alternative.  As  quality  increased,  the 
alternative  became  more  valuable  to  the  organization  conducting  the  C4I  SoS  evaluation, 
because  evaluations  become  more  accurate,  more  effective,  and  more  efficient.  The  SSF 
used  to  score  this  EM  is  illustrated  in  Figure  39. 
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Quality  of  Planning  Outputs 


Figure  39.  Utility  Function  for  Quality  of  Planning  Outputs. 

This  is  the  stakeholder  validated  Utility  Function  used  for  this  EM.  A  raw  score  of  4.0  is 

ideal  and  received  a  utility  of  1 .0.  SSF  3  Parameters  were  L  =  0,  B  =  2.0,  U  =  4.0,  D  (U  - 

L)  =  4.0,  and  S  =  0.5. 

5.  Elasticity  of  Labor  and  Elasticity  of  Duration  Utility  Curves 

Elasticity  of  Labor  was  an  EM  that  reported  on  how  the  alternative  responded  to 
changes  in  the  C4I  SoS  under  evaluation.  Specifically,  the  percent  change  in  labor  hours 
to  conduct  planning  phase  of  JC3M  was  divided  by  the  percent  change  in  input 
parameters  (number  of  SUT,  number  of  new  capabilities,  number  of  old  capabilities)  and 
the  resulting  ratio  was  the  measure  of  elasticity. 

Values  less  than  1.0  are  inelastic,  and  indicated  that  changes  in  the  C4I  SoS  under 
evaluation  cause  a  change  in  the  number  of  labor  hours  required  at  a  smaller  rate.  As  the 
SoS  increased  in  size,  there  was  less  labor  required,  per  system,  to  evaluate  the 
performance  of  the  SoS.  Elasticity  is  valuable  when  conducting  evaluations  of  C4I  SoS 
as  large  or  larger  than  the  candidate  test  architecture  described  in  Chapter  IV  section  C.2, 
Select  a  Candidate  C4I  Test  Architecture.  The  SSF  used  to  score  Elasticity  of  Labor  is 
illustrated  in  Figure  40. 

Elasticity  of  Duration,  also  illustrated  in  Figure  40  is  an  equivalent  EM,  and 
measured  how  the  duration  (number  of  days)  of  the  alternative  process  changed  in 
response  to  changes  in  the  C4I  SoS  under  evaluation.  Elasticity  of  Duration  would  be 
valuable  when  conducting  evaluations  of  C4I  SoS  as  large  or  larger  than  the  candidate 
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test  architecture  described  in  Chapter  IV  section  C.2,  Select  a  Candidate  C4I  Test 
Architecture. 


Elasticity  of  Labor/Duration 


Figure  40.  Utility  Function  for  Elasticity  of  Labor  and  Elasticity  of 
Duration. 

This  is  the  stakeholder  validated  Utility  Function  used  for  this  EM.  A  lower  number  for 
Elasticity  was  desirable.  Values  less  than  0.5  days  received  a  utility  of  1.0.  Values 
greater  than  or  equal  to  1.5  days  receive  a  utility  of  1.0.  SSF  3  Parameters  were  L  =  0.5, 

B  =  1.0,  U  =  1.5,  D  (U  —  L)  =  1.0,  and  S  =  2.0. 

Values  less  than  1.0  are  inelastic,  and  indicated  that  changes  in  the  C4I  SoS  under 
evaluation  cause  a  change  in  the  number  of  days  required  at  a  smaller  rate.  As  the  SoS 
increased  in  size,  there  were  less  days  required,  per  system,  to  conduct  planning  of  the 
SoS  test.  Elasticity  is  valuable  when  conducting  evaluations  of  C4I  SoS,  to  determine  the 
effects  that  increases  in  the  number  of  systems  will  have  on  the  duration  for  planning  the 
SoS  evaluation.  Chapter  IV  section  C.2,  described  the  baseline  candidate  C4I  test 
architecture;  by  adding  or  reducing  the  number  of  systems  one  can  determine  the 
elasticity  of  the  number  of  days  required  to  plan  a  C4I  SoS  event. 

E.  ANALYTICAL  HIERARCHY  PROCESS 

Buede  [2000:  369]  described  several  methods  to  elicit  weights  for  calculating  the 
relative  worth  of  EM.  The  team  chose  the  AHP  because  it  is  useful  for  cases  where  there 
are  3-7  objectives  [Saaty,  1994], 

The  weights  were  elicited  from  three  SMEs,  using  the  AHP  verbal  mode  for 
pairwise  comparisons.  The  SMEs  provided  their  assessment  of  how  the  EMs  compared 
against  each  other  using  the  verbal  scale  and  the  numerical  scale  in  Figure  41. 
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Equal  Weakly 


STRONGLY  StronGLY  AbSOLUTELY 


Figure  41.  Sample  Analytical  Hierarchy  Process  Scale. 

This  scale  ties  the  numerical  AHP  values  to  the  verbal  equivalents  utilized  during  the 
JC3M  stakeholder  questionnaire.  This  sample  scale  ranges  illustrates  the  range  of 
potential  values  from  one  ninth  as  valuable  to  nine  times  as  valuable. 

A  value  of  1  signifies  the  objectives  were  equal;  3  signifies  an  objective  was 

weakly  superior  to  the  other  objective;  5  signifies  an  objective  was  strongly  superior;  7 

signifies  very  strongly;  and  9  signifies  absolutely  superior.  Objective  measures  are 

compared  pairwise,  i.e.  Percentage  of  Traceable  Measures  is  strongly  superior  (5)  to 

Days  to  Plan  Evaluation.  The  AHP  then  attributed  a  relative  weight  of  5  to  Percent 

Traceable  Measures  and  a  relative  weight  of  1  to  Days  to  Plan  Evaluation.  The  reciprocal 

comparison  is  also  recorded  in  the  cell  on  the  opposite  side  of  the  diagonal;  Days  to  Plan 

Evaluation  is  1/5  with  respect  to  Percentage  of  Traceable  Measures.  The  relative  weights 

remain  the  same;  it  is  only  the  order  of  measures  that  changes. 

This  assessment  resulted  in  some  inconsistent  weighting  of  the  objectives.  This  is 
not  too  unusual  for  the  first  iteration  in  trying  to  quantify  the  relative  importance  of  the 
different  evaluation  measures.  As  this  inconsistency  was  discovered  after  the  SMEs  were 
no  longer  available,  the  team  did  not  have  the  opportunity  to  re-engage  the  SMEs  in  order 
to  resolve  those  inconsistencies.  The  degree  of  inconsistency  of  the  pairwise  comparison 
matrix  exceeded  0.1.  Nevertheless,  the  team  performed  an  eigenvector  calculation  on  the 
responses.  An  eigenvector  is  a  mathematical  term  encountered  when  studying  linear 
transformations.  In  AHP,  the  result  of  the  eigenvector  calculation  provides  the  weights 
for  each  EM.  Saaty  [1994]  demonstrated  mathematically  that  the  eigenvector  solution 
was  the  best  approach  to  obtain  weights  from  the  AHP  matrix.  Equation  4  can  be  treated 
as  an  eigenvector  problem  and  is  solved  for  w  to  obtain  the  weights. 
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Aw  =  nw  (4) 

where  A  is  the  AHP  matrix,  w  is  the  eigenvector  and  Wj  is  the  weight  of  EMi,  and 
n  is  the  dimension  of  the  matrix  and  the  eigenvector.  Equation  4  is  the  Eigenvalue 
equation  expressed  as  a  matrix. 
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In  Equation  5  solving  for  w  yields  the  weights  for  each  evaluation  measure 
[Ishikawa,  2007],  shown  in  Table  15. 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 
of  Labor 

Elasticity 

of 

Duration 

EM 

Weights 

0.248 

0.058 

0.419 

0.084 

0.192 

Table  15.  AHP  Evaluation  Measure  Weights. 

The  weights  are  calculated  using  an  iterative  eigenvector  process. 


A  sensitivity  analysis  on  the  effect  of  the  weights  was  performed.  The  conclusion 
is  that  the  noted  minor  inconsistencies  do  not  invalidate  the  final  results.  Based  on  sound 
reasoning,  the  results  of  applying  these  weights  to  the  utility  scores  have  resulted  in  valid 
relative  rankings  of  the  alternatives  even  though  the  absolute  values  used  to  populate  the 
quantitative  decision  matrix  would  differ  slightly  if  the  weights  were  changed. 

F.  RAW  DATA  VALUES 

This  section  summarizes  the  raw  data  values  obtained  through  modeling  and 
simulation  and  offline  evaluation. 

1.  Results  of  Percent  Traceable  Measures  EM 

Percentage  of  Traceable  Measures  (PTM)  was  the  EM  for  the  Define  Measures 
(1.3.2)  function  of  the  JC3M  value  hierarchy.  The  objective  of  the  Define  Measures 
function  was  to  determine  how  well  an  alternative  identified  measures  of  performance 
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(MOP)  when  evaluating  the  SoS.  An  EM  should  not  be  confused  with  a  MOP.  EMs  are 
measures  created  by  the  team  to  gauge  functions  of  the  JC3M  system.  MOPs  are 
measures  that  gauge  performance  of  a  C4I  SoS.  The  detailed  methodology  used  to  derive 
the  PTM  EM  is  included  in  Appendix  G.  The  results  are  also  summarized  in  this  section. 

Definition  of  PTM: 


This  EM  was  calculated  by  dividing  the  number  of  measures  (traceable  to 

stakeholder  requirements)  generated  by  the  alternative,  by  the  number  of  measures 

generated  by  the  alternative 

,  #  Traceable  Measures  Created 

PTM  = - 

#  Total  Measures  Created 


(6) 


However,  the  team  came  to  the  conclusion  that  it  was  not  feasible  to  calculate  the 
PTM  EM  as  it  was  defined  because  that  would  entail  exercising  each  of  the  alternative 
systems  and  developing  MOPs  for  each  alternative.  Therefore,  a  proxy  measure  was 
developed  that  could  serve  as  an  indicator  to  the  performance  of  the  Define  Measures 
function. 

Proxy  Definition  of  PTM: 

This  EM  was  calculated  by  taking  the  number  of  authoritative  sources  that  were 
considered  in  the  process  and  then  dividing  by  the  total  number  of  authoritative  sources 
available  for  the  SoS. 

,  #  Authoritative  Sources  Considered 

PTM  = - 

#  Authoritative  Sources  Available  ,n 


The  concept  was  that  analysts  performing  the  Define  Measures  function  should 
consider  all  available  documentation  for  the  SoS.  Considering  all  available 
documentation  helped  to  ensure  that  all  requirements  and  testable  capabilities  are 
captured  in  the  process  and  subsequently  MOPs  are  defined  for  each.  The  team 
considered  that  in  reviewing  a  wider  set  of  authoritative  sources  the  process  would  yield  a 
higher  number  of  requirements  and  capabilities,  and  in  turn  provide  more  traceable 
metrics. 
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If  an  alternative  considered  every  document  in  the  comprehensive  list  of 
authoritative  documents,  as  discussed  in  Appendix  G,  it  received  a  score  of  100%.  Table 
26,  in  Appendix  G,  contains  the  comprehensive  list  of  authoritative  documents,  weights 
for  each  document,  and  the  score  for  each  alternative.  If  an  alternative  uses  a  document 
in  its  process  then  the  alternative  receives  the  complete  score  for  that  document.  The 
final  results  of  the  PTM  EM  is  included  in  Table  15. 

2.  Results  of  Quality  of  Planning  Outputs  EM 

Quality  of  Planning  Outputs  was  the  EM  for  the  Planning  Results  (1.4.3)  function 
of  the  JC3M  value  hierarchy.  The  objective  of  the  Planning  Results  function  was  to 
determine  how  well  an  alternative  produced  planning  documents  when  planning  for  a  C4I 
SoS  evaluation.  This  EM  was  calculated  by  assigning  an  overall  quality  level  to  the 
planning  documents  produced  by  the  alternative.  The  detailed  methodology  used  to 
derive  the  Quality  of  Planning  Outputs  EM  is  included  in  Appendix  C.  However,  the 
results  are  also  summarized  here.  The  results  of  the  Quality  of  Planning  Outputs  EM  are 
included  in  Table  16. 

The  JC3M  team  decided  to  use  a  Likert  scale  to  determine  the  quality  of  planning 
products  evaluation  measure.  The  Likert  scale  was  set  from  1  to  4  with  four  being  the 
best  score  and  one  being  the  worst  score. 

The  team  assembled  a  questionnaire  (provided  at  Appendix  C)  that  reviewed  the 
quality  of  planning  products.  Each  question  also  contained  a  criterion  in  order  to  help  the 
respondent  differentiate  between  a  best  value  of  4  or  a  worst  value  of  1 . 

After  creating  the  questionnaire  the  team  solicited  SME  input.  One  SME  from 
MCTSSA,  and  two  from  China  Lake,  answered  the  questionnaire. 

The  raw  score  was  calculated  by  averaging  the  scores  for  each  alternative  from 
the  questions  answered  by  each  of  the  SMEs. 
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3.  Compiled  results  for  all  EMs 

Table  16  depicts  the  results  including  both  M&S  and  offline  evaluation,  which 
provided  values  to  express  the  performance  of  the  alternatives  for  every  critical  EM. 
(Cost  is  calculated  separately.) 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity  of 
Labor 

Elasticity 
of  Duration 

(%) 

(Days) 

(1-4  Likert 
Scale) 

(unit  less) 

(unit  less) 

Ideal  Value 

100% 

Less  is 
better 

4  is  Ideal 

Less  is 
better 

Less  is 
better 

FEDOS 

0 

140 

3.17 

0.87 

0.87 

MC3T 

72 

121 

3.25 

0.78 

0.78 

JTEM  CTM 

92 

73 

3.42 

1.04 

0.83 

SCR 

92 

158 

3.00 

0.98 

0.98 

FCB 

88 

127 

2.75 

0.72 

0.72 

Table  16.  Raw  Data  Matrix. 


This  is  the  Raw  Data  Matrix  with  the  raw  scores  for  each  EM.  The  results  are  the  raw 
scores  that  are  still  in  the  units  of  the  respective  EM.  Scores  for  all  alternatives  were 
rounded  for  display  in  this  table. 


G.  CALCULATING  AN  OVERALL  UTILITY  SCORE 

As  stated  in  Section  B  “Multi  Attribute  Utility  Theory”  of  this  Chapter,  the  EMs 
are  combined  into  a  single  measure  for  each  alternative  called  the  overall  utility  score. 
Equation  2,  the  value  function  equation,  was  used  to  calculate  the  overall  utility  score. 
Table  17  and  Table  18  depict  the  intermediate  and  final  values  obtained  using  the 
equation. 
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Table  17  depicts  the  EMs  after  transforming  the  raw  score  into  the  utility  score 
using  the  Wymorian  SSFs.  A  utility  score  of  1.0  is  ideal. 


Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity  of 
Labor 

Elasticity 
of  Duration 

Range 

0-1 

0-1 

0-1 

0-1 

0-1 

FEDOS 

0.00 

0.66 

0.92 

0.75 

0.75 

MC3T 

0.10 

0.88 

0.94 

0.86 

0.86 

JTEM  CTM 

0.98 

1.00 

0.96 

0.42 

0.80 

SCR 

0.98 

0.37 

0.89 

0.54 

0.54 

FCB 

0.90 

0.83 

0.82 

0.92 

0.92 

Table  17.  Utility  Score  Data  Matrix. 

This  is  the  Utility  Score  Data  Matrix  with  the  utility  scores  for  each  EM.  The  results  are 
the  individual  utility  scores  that  range  from  0  to  1,  where  1  is  ideal.  Scores  for  all 
alternatives  were  rounded  for  display  in  this  table. 

After  obtaining  the  utility  scores,  the  scores  are  then  multiplied  by  their  respective 
weights.  Finally,  the  overall  utility  score  is  obtained  by  adding  the  individual  EM 
weighted  utility  scores  for  each  alternative.  The  individual  EM  weighted  utility  scores, 
overall  utility  scores,  and  LCCE  (in  millions  of  dollars)  for  each  alternative  are  recorded 
in  Table  18. 
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Percentage 

of 

Traceable 

Measures 

Days  to 
Plan 

Evaluation 

Quality  of 
Planning 
Outputs 

Elasticity 
of  Labor 

Elasticity  of 
Duration 

Overall 

Utility 

(0-1) 

LCCE 

($  Mil) 

FEDOS 

0.00 

0.04 

0.39 

0.06 

0.14 

0.63 

5.01 

MC3T 

0.02 

0.05 

0.39 

0.07 

0.17 

0.71 

5.98 

JTEM  CTM 

0.24 

0.06 

0.40 

0.04 

0.15 

0.89 

6.97 

SCR 

0.24 

0.02 

0.37 

0.05 

0.10 

0.79 

7.72 

FCB 

0.22 

0.05 

0.34 

0.08 

0.18 

0.87 

8.13 

Table  18.  Quantitative  Modeling  Decision  Matrix. 

This  table  displays  the  estimated  performance  of  each  alternative  as  weighted  utility 
scores  for  each  EM.  The  weighted  utility  scores  are  summed  and  displayed  adjacent  to 
the  LCCE  (in  millions)  in  the  yellow  columns.  Weighted  Utility  and  Overall  Utility 
scores  for  all  alternatives  were  rounded  for  display  in  this  table;  these  rounded  values 
were  used  for  Figure  42  and  Figure  43. 

H.  UTILITY  SCORE  VERSUS  LIFE  CYCLE  COST  ESTIMATE 

The  final  step  of  AoA  was  to  compare  alternatives  side  by  side  using  a  Utility 
versus  Life  Cycle  Cost  Estimate  Plot  (see  Figure  42  and  Figure  43).  The  plot  allowed 
decision  makers  to  visually  compare  the  cost  of  each  alternative,  the  estimated 
performance  of  each  alternative,  and  the  cost  to  achieve  estimated  performance  in 
specific  alternatives  simultaneously.  This  data  was  utilized  in  the  Final  Recommendation 
phase,  where  a  summary  of  the  project  results  was  provided  for  stakeholder 
consideration.  Figure  42  and  Figure  43  are  similar;  Figure  43  is  scaled  to  increase  the 
ability  to  differentiate  the  relative  cost  and  performance  of  each  alternative. 

FCB  has  the  highest  LCCE  at  $8. 13M.  SCR  is  next  at  S7.72M,  followed  by 
ITEM  CTM  which  costs  $6.97M.  FEDOS  and  MC3T  follow  with  LCCE  of  $5.01M  and 
$5.97M  respectively.  JTEM  CTM  and  FCB  dominate  all  of  the  other  alternatives  for 
utility  with  scores  of  0.89  and  0.87.  JTEM  CTM  and  FCB  have  virtually  equal  utility,  yet 
the  JTEM  CTM  LCCE  costs  approximately  $1.16M  less  than  FCB.  JTEM  CTM  has  the 
median  LCCE  of  the  five  alternatives  which,  coupled  with  the  highest  utility,  clearly 
dominated  all  other  alternatives. 
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Utility 


Utility  versus  Life  Cycle  Cost  Estimate 
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Figure  42.  Utility  versus  Life  Cycle  Cost  Estimate  Plot. 

This  plot  illustrates  the  utility  of  each  alternative  while  displaying  the  estimated  cost  to 
achieve  that  performance.  JTEM  CTM  utility  slightly  exceeded  that  of  the  FCB 
alternative,  with  scores  of  0.89  and  0.87  respectively.  JTEM  CTM  and  FCB  utility  scores 
are  followed  in  descending  order  by  SCR,  MC3T,  and  FEDOS.  ECCE  values  (in 
millions)  range  from  FEDOS  at  $5.01  to  FCB  at  $8.13.  JTEM  CTM  has  the  median 
LCCE  value  at  $6.97. 
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Utility  versus  Life  Cycle  Cost  Estimate 


Life  Cycle  Cost  Estimate  ($MIL) 


Figure  43.  Utility  versus  Life  Cycle  Cost  Estimate  Plot  Zoom. 

This  plot  illustrates  the  utility  of  each  alternative  while  displaying  the  estimated  cost  to 
achieve  that  performance.  FEDOS  had  the  lowest  estimated  utility,  followed  by  MC3T, 
SCR,  FCB,  and  JTEM  CTM.  JTEM  CTM  and  FCB  scored  0.89  and  0.87  respectively. 
FEDOS  had  the  lowest  estimated  cost,  followed  by  MC3T,  JTEM  CTM,  SCR,  and  FCB. 
JTEM  CTM  slightly  exceeds  the  other  alternatives  for  utility,  and  has  the  median  cost  of 
the  five  alternatives.  This  figure  differs  from  Figure  43  in  that  the  scale  is  revised,  with 
the  origin  at  $4.5M,  0.55  utility  in  order  to  more  clearly  demonstrate  the  differences 
between  alternatives. 
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VII.  FINAL  RECOMMENDATIONS  AND  CONCLUSIONS 


A.  PERFORMANCE  ANALYSIS 

The  JC3M  team  determined  the  highest  performance  was  exhibited  by  the  JTEM 
CTM  alternative,  at  an  overall  utility  rating  of  0.89  on  a  1-0  scale.  In  order  of  decreasing 
utility,  the  other  alternatives  were  FCB  at  0.87;  SCF  at  0.79;  MC3T  at  0.71;  and  FEDOS 
at  0.63. 

The  team’s  confidence  in  the  accuracy  of  the  performance  evaluations  differed 
among  the  alternatives.  The  team  had  very  high  confidence  in  the  accuracy  of  the 
performance  evaluation  of  MC3T  and  FEDOS.  The  team  had  less,  although  still  high, 
confidence  in  the  accuracy  of  the  JTEM  CTM  performance  evaluation.  The  team  had 
less  confidence  in  the  accuracy  of  the  performance  evaluation  of  FCB  and  SCR. 

Of  all  the  alternatives  considered,  the  team  had  the  most  confidence  in  the 
estimates  of  performance  calculated  for  both  MC3T  and  FEDOS.  These  are  the  only 
two  alternatives  that  had  been  utilized  in  the  conduct  of  full-scale  C4I  SoS  evaluations. 
This  real  world  use  of  the  two  alternatives  (MC3T  and  FEDOS)  meant  the  team  was  able 
to  review  historical  records  of  actual  inputs  and  outputs  of  the  alternatives,  and  observe 
the  duration  of  each  of  these  alternatives.  Given  the  number  of  data  sources  and  recent 
nature  of  this  data,  while  MC3T  was  implementing  their  processes  as  the  JC3M  team  was 
analyzing  alternative  solutions,  the  team  had  confidence  in  the  accuracy  of  the  estimate  of 
performance  from  these  two  alternatives.  The  team  has  observed  first  hand  the  type, 
scope,  and  duration  of  efforts  involved  in  MC3T,  which  are  reflected  in  the  overall  utility 
score  of  the  alternative,  and  concluded  MC3T  will  deliver  better  utility  than  FEDOS. 

While  the  JTEM  CTM  has  not  been  used  for  full  scale  C4I  SoS  evaluations,  it  is 
the  alternative  in  which  the  JC3M  team  had  the  next  greatest  confidence  in  the  accuracy 
of  the  performance  evaluation.  The  JTEM  CTM  has  been  reviewed  by  a  very  large  group 
of  stakeholders,  the  JTEM  Community  of  Interest  (COI).  These  C4I  SoS  experts  have 
provided  input  to  the  JTEM  CTM  methods  and  processes.  A  subset  of  the  COI 
participated  in  the  JTEM  “Rock  Drills”  which  included  a  series  of  tabletop  exercises  used 
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to  evaluate  and  refine  the  JTEM  CTM.  JTEM  CTM  members  have  participated  in  the 
definition  and  development  of  JC3M.  They  have  explicitly  validated  the  content  and 
definition  of  the  subset  of  JTEM  CTM  tasks  which  were  identified  as  a  JC3M  alternative 
solution.  This  validation  included  a  review  of  the  scope,  tasks,  components,  duration, 
and  resources  required  for  JTEM  CTM  to  perform  a  JC3M  C4I  SoS  evaluation.  Because 
of  the  COI  review,  Rock  Drills,  and  performance  validation  through  modeling  of  the 
JTEM  CTM;  the  JC3M  team  has  confidence  in  the  accuracy  of  the  estimate  of 
performance  of  this  alternative,  albeit  at  a  lesser  level  than  the  real-world  performance  of 
both  FEDOS  and  MC3T. 

The  team  had  less  confidence  in  the  accuracy  of  the  performance  evaluation  of  the 
SCR  and  FCB  alternatives.  Neither  of  these  alternatives  benefited  from  the  very 
extensive  C4I  SoS  SME  review  which  supported  JTEM  CTM.  Additionally,  neither 
alternative  has  been  used  in  real  world  SoS  evaluations,  unlike  FEDOS  and  MC3T. 

B.  COST  ANALYSIS 

The  JC3M  team  determined  the  least  expensive  alternative  was  the  FEDOS 
alternative,  at  a  life  cycle  cost  of  S5.01M  over  the  ten-year  projected  lifespan  of  JC3M. 

In  order  of  increasing  cost,  the  other  alternatives  were  MC3T  at  $5.98M;  JTEM  CTM  at 
S6.97M;  SCR  at  $7.72M;  and  FCB  at  $8.13M. 

The  team  noted  that  MC3T  was  estimated  to  cost  approximately  $970,000  more 
than  FEDOS,  which  it  has  replaced,  over  the  ten  year  span  of  the  LCCE.  Because  the 
LCCE  postulated  the  conduct  of  a  single  C4I  SoS  evaluation  per  year  through  the  eight 
year  O&S  phase,  this  cost  difference  equates  to  approximately  $121,000  per  C4I  SoS 
evaluation. 

The  team  also  noted  that  FCB  is  approximately  $410,000  more  than  SCR. 

Unique  to  FCB  is  the  use  of  senior  SMEs  who  generate  the  performance  measures,  yet 
are  not  part  of  the  organization  conducting  the  C4I  SoS  evaluation.  The  team  calculated 
the  cost  of  senior  SMEs,  and  included  these  as  an  overall  cost  to  DoD.  While  the  cost  of 
their  labor  was  not  charged  to  C4I  test  organizations,  the  use  of  these  resources  is  a  cost 
to  DoD.  If  the  cost  of  only  personnel  organic  to  the  test  organization  had  been  reported 
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for  FCB,  the  cost  would  have  been  lower  by  $1.14M  but  this  would  have  represented 
only  part  of  the  cost  of  DoD  resources. 

In  a  discussion  with  the  Deputy  Chief  of  the  Methods  and  Processes  Division 
[Wilson,  2007],  the  team  reviewed  the  maturity  of  the  JTEM  CTM  and  its  readiness  for 
implementation.  While  JTEM  plans  to  deliver  two  more  versions  of  the  CTM  by  2009, 
the  development  between  2007  and  2009  is  primarily  focused  on  development  of  the 
Joint  Mission  Environment.  Wilson  [2007]  and  the  JC3M  team  lead  agreed  that  the  CTM 
was  robust  in  its  current  state,  but  would  require  additional  development. 

The  cost  to  complete  development  of  the  JTEM  CTM  [Bjorkman  (b),  2007],  is 
approximately  $3.5M  through  2009.  Although  the  team  included  the  complete 
development  cost  of  the  JTEM  CTM  in  the  LCCE,  they  recognized  that  development  is  a 
non-recurring  cost  that  is  not  borne  by  any  C4I  test  organization. 

O&S  is  the  most  important  cost  from  the  perspective  of  C4I  test  organizations. 
O&S  is  a  recurring  cost,  borne  by  each  C4I  test  organization  that  implements  an 
alternative,  for  each  year  that  alternative  is  used.  With  the  exception  of  the  JTEM  CTM 
development  cost  ($3.5M),  O&S  is  the  largest  portion  of  the  LCCE  for  each  alternative. 
The  JTEM  CTM  alternative  had  the  lowest  overall  O&S  cost  at  $2.25M  over  the  ten  year 
LCCE.  This  O&S  cost  was  the  lowest  of  all  alternatives  by  $1.66M. 

Of  all  the  alternatives  considered,  the  team  had  the  most  confidence  in  the  cost 
estimates  of  both  MC3T  and  FEDOS.  The  team  was  able  to  review  historical  records  of 
the  planned  and  actual  costs  of  each  of  these  alternatives:  the  confidence  in  these  cost 
estimates  was  high.  The  team  had  the  next  highest  confidence  in  the  cost  of  the  JTEM 
CTM  alternative.  The  team  validated  their  cost  model  with  the  JTEM  and  included  the 
actual  funding  required  to  complete  the  JTEM  CTM  development  in  the  LCCE.  The 
team  had  less  confidence  in  the  cost  estimate  of  the  SCR  and  FCB  alternatives.  Neither 
of  these  alternatives  have  been  used  in  real  world  SoS  evaluations,  unlike  FEDOS  and 
MC3T.  Additionally,  the  models  for  SCR  and  FCB  are  simply  estimates  and  have  not 
been  validated  as  in  the  case  of  JTEM  CTM. 
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C.  PERFORMANCE  VERSUS  COST  ANALYSIS 

As  stated  previously,  the  JC3M  team  determined  the  performance  and  cost  of 
each  alternative  to  be  as  follows:  JTEM  CTM  at  0.89  and  $6.97M,  FCB  at  0.87  and 
$8.13M;  SCR  at  0.79  and  $7.72M;  MC3T  at  0.71  and  $5.98M;  and  FEDOS  at  0.63  and 
5.01M. 

The  team  was  interested  to  observe  stratification  in  performance  ratings;  MC3T 
and  FEDOS  were  relatively  close  in  both  cost  and  utility;  FCB  and  SCR  were  similarly 
close;  and  all  four  of  these  alternatives  can  be  seen  to  deliver  increased  utility  in  return 
for  increased  cost. 


The  slope  of  the  line  expressing  the  relationship  between  cost  and  utility  is 
0.070847,  where  a  change  in  cost  results  in  a  change  in  utility.  The  equation  for 
calculating  the  slope  of  the  regression  line  b  is  given  by 


(8) 


where  x  represents  the  cost  value  of  a  specific  alternative,  and  x  the  average  of  the  cost 


values,  where  y  represents  the  cost  value  of  a  specific  alternative,  and  y  the  average  of 
the  utility  values  [Baker,  2006:  6].  The  regression  line  (Fi)  can  be  seen  in  Figure  44. 


Had  the  team  considered  alternatives  as  a  portfolio,  and  combined  attributes  (the 
C2  SME  panel  from  the  FCB  alternative,  for  example,  with  the  MC3T  use  of  Service 
stakeholders)  it  would  have  been  possible  to  define  mixed-attribute  alternatives  that 
provided  more  utility  for  the  same  cost.  The  most  cost-effective  combinations  would 
have  formed  a  curved  efficient  frontier  [Gunser,  2004:  26]  of  utility  for  every  level  of 
cost.  Because  combinations  were  not  considered,  a  linear  efficient  frontier  (L2)  is  defined 
by  the  three  more  efficient  alternatives  and  is  seen  in  Figure  44.  Given  their  respective 
utility  scores,  both  the  FCB  and  SCR  alternatives  are  to  the  right  of  the  frontier,  less 
efficient  than,  and  dominated  by,  the  JTEM  CTM  alternative. 
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Cost  vs.  Utility 


Figure  44.  Cost  Versus  Utility  with  Trendline. 

The  figure  illustrates  estimated  LCCE  and  performance  values  for  the  five  alternatives. 

The  LCCE  and  performance  values  for  four  alternatives  (FEDOS,  MC3T,  SCR,  and 
FCB)  were  used  to  generate  a  trendline  to  predict  performance  based  on  alternative  cost. 

With  this  trendline,  the  predicted  performance  of  JTEM  CTM  is  greater  than  what  would 
be  expected,  based  on  the  JTEM  CTM  LCCE.  The  JTEM  CTM  alternative  is  above  the 
predicted  performance  value  (Ld,  indicating  it  has  a  higher  value  to  cost  ratio  than  other 
alternatives.  An  efficient  frontier  (L2)  depicts  the  relative  inefficiency  of  FCB  and  SCR 
when  compared  to  JTEM  CTM. 

D.  CONCLUSION 

The  JTEM  CTM  alternative  dominated  both  FCB  and  SCR,  providing  higher 
estimated  utility  for  less  cost.  JTEM  CTM  exceeded  the  performance  of  all  alternatives 
by  a  very  slight  margin.  Though  the  team  had  different  levels  of  confidence  in  the  utility 
scores  of  each  alternative,  they  are  confident  that  the  results  and  conclusions  are  valid. 

As  described  in  section  A,  the  team  had  more  confidence  in  the  JTEM  CTM  utility  score 
than  in  the  next  highest  alternative,  FCB. 

The  JTEM  CTM  had  the  median  LCCE,  and  the  lowest  O&S  cost  by  a  large 
margin.  Chapter  V  section  A.2.b  “Operations  and  Support  Costs”  noted  that  costs  in  the 
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O&S  phase  were  primarily  driven  by  the  use  of  personnel  resources.  The  73  day  duration 
of  the  JTEM  CTM  planning  phase  led  to  the  smallest  O&S  cost.  The  next  alternative, 
MC3T,  had  a  121  day  planning  phase,  with  correspondingly  higher  O&S  costs.  This  is 
significant  because  O&S  is  a  recurring  cost,  borne  by  every  C4I  test  organization  that 
utilizes  an  alternative.  Development  of  JTEM  CTM  is  the  largest  portion  of  the  LCCE 
for  this  alternative.  However,  the  development  of  JTEM  CTM  is  a  nonrecurring  cost 
borne  by  OSD  and  not  borne  by  any  C4I  test  organization. 

The  team  recommends  monitoring  the  development  of  the  JTEM  CTM  for  further 
maturation.  The  JTEM  CTM  promises  significant  improvements  in  the  overall  utility  of 
C41  SoS  evaluations,  at  a  significantly  reduced  operating  cost,  and  deserves  further 
investigation.  The  scope  of  this  project,  however,  does  not  allow  that  investigation.  The 
team  recommends  detailed  investigation  of  the  JTEM  CTM  in  its  entirety,  and  optimizing 
of  personnel  resources  and  organizations  with  a  modeling  tool  such  as  POW-ER.  The 
team  also  recommends  for  JTEM  CTM  to  conduct  a  C41  SoS  evaluation  as  soon  as 
feasible,  to  validate  the  JTEM  CTM  methods  and  processes  in  a  “real  world”  event. 

E.  AREAS  FOR  FURTHER  STUDY 

The  JC3M  team  identified  several  SoS  issues  in  the  course  of  the  project,  and 
recommended  investigation  of  selected  issues  when  additional  time  is  available. 

1.  Establish  a  Manager 

The  team  believed  the  C41  acquisition  and  testing  communities  would  benefit 
from  a  dedicated  Joint  C41  SoS  manager.  A  dedicated  SoS  manager  could  provide 
consistency  of  knowledge  in  an  evolving  C41  acquisition  and  testing  environment.  Their 
role  could  include,  but  should  not  be  limited  to,  the  following  tasks:  Documenting  C4I 
SoS  capabilities,  long  range  SoS  capabilities  planning,  testing  requirements  management, 
reducing  SoS  testing  cycle  and  costs,  assessing  the  improvements  in  SoS  processes, 
supporting  developmental  and  operational  testing  as  a  stakeholder,  actively  participating 
in  IPTs  that  address  SoS  testing  issues,  risk  management,  and  addressing  ad  hoc  SoS 
configuration  resulting  from  new  threats  and  concepts.  This  is  consistent  with  the 
concepts  of  capability  portfolio  management  under  the  Joint  capabilities  area  construct. 
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2.  Initiate  Risk  Management  Strategies 

The  C4I  SoS  continues  to  change  configuration  as  it  exhibits  new  capabilities. 
The  continually  changing  architecture  of  the  C4I  SoS  increases  the  probability  of 
capability  failures.  The  increasing  probability  of  capability  failures  creates  risks  that 
need  be  managed.  The  JC3M  team  believes  risk  management  strategies  should  be 
developed  and  applied  to  the  C4I  SoS.  The  JC3M  team  has  compiled  a  preliminary  list 
of  risks  that  should  be  managed  across  the  C4I  SoS,  including  the  lack  of  a  single  entity 
responsible  for  SoS  performance;  lack  of  an  objective,  repeatable,  and  methodical 
approach  to  address  individual  system  problems  that  impact  SoS  functionality;  varying 
levels  of  maturity  of  critical  systems  within  the  C4I  SoS  architecture;  lack  of  consistent 
SoS  integration  of  individual  systems;  and  varied  interfaces  between  individual  systems 
that  comprise  the  C4I  SoS. 

3.  Modify  the  Acquisition  Process 

Systems  that  are  components  of  the  C4I  SoS  have  their  capabilities  defined  as  if 
they  exist  in  a  vacuum,  and  their  impact  on  C4I  SoS  capabilities  is  not  considered. 
System  level  capabilities  should  be  considered  in  light  of  their  effect  on  the  C4I  SoS, 
which  is  consistent  with  the  concept  of  capability  portfolio  management.  The  DoD  C4I 
SoS  acquisition  process  should  require  component  system  sponsors  to  define  C4I  SoS 
level  effects  when  creating  CDDs  and  CPDs;  establish  a  funding  process  for  SoS  testing; 
require  systems  to  identify  their  effects  on  the  C4I  SoS  before  fielding;  include  end-to- 
end  SoS  effectiveness  testing  as  an  explicit  part  of  Operational  Testing 
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APPENDIX  A.  PROJECT  PLAN 


INTRODUCTION 

This  is  the  Project  Management  Plan  (PMP)  for  the  capstone  project  to  be  completed  by 
the  Marine  Corps  Tactical  Systems  Support  Activity  (MCTSSA)  Cohort  in  the  Naval 
Postgraduate  School  (NPS)  Masters  of  Science  in  Systems  Engineering  program.  The 
Joint  Command,  Control,  Communications,  Computers,  and  Intelligence  (C4I)  Capability 
Certification  Management  (JC3M)  project  will  create  a  system  for  certifying  the 
capability  of  a  C4I  System  of  Systems  (SoS).  A  System  of  Systems  in  the  context  of  this 
project  is  a  group  of  individual  C4I  systems,  which  may  not  have  been  designed, 
acquired,  or  managed  as  a  collective  enterprise,  but  are  being  put  together  as  such  and 
forming  an  interdependent  entity.  The  JC3M  system  is  intended  to  perform  an 
assessment  that  will  identify  the  desired  war  fighting  capabilities  and  ensure  that  the  SoS 
under  test  meets  these  requirements  in  the  intended  environment. 

PROJECT  DESCRIPTION 

Across  the  Department  of  Defense  (DoD),  early  C4I  systems  were  designed,  acquired, 
and  fielded  independently,  each  addressing  a  single  warfighting  function,  such  as 
logistics,  fire  support,  or  intelligence.  Over  time,  warfighting  has  grown  in  complexity, 
tempo,  and  scope,  so  organizations  must  be  able  to  respond  with  increased  agility  across 
greater  distances.  This  complexity  is  compounded  by  adaptive  and  elusive  adversaries. 

To  combat  today’s  adversaries,  DoD  forces  fight  jointly. 

Individual  C4I  systems,  which  may  not  have  been  designed,  acquired,  or  managed  as  a 
collective  enterprise,  are  being  put  together  as  such  and  forming  an  interdependent  entity, 
a  SoS.  Today’s  C4I  SoS,  whether  Joint  or  Single  Service,  is  required  to  transport  and 
process  shared  data  throughout  the  operating  forces.  Problems  are  abundant  because  there 
is  no  baseline,  standard  configuration,  or  overall  management  of  the  SoS. 

C4I  system-level  acquisition,  testing,  and  management  are  well  understood,  and 
individual  systems  have  performance  requirements.  However,  ever-changing 
configurations  of  C4I  SoS  may  not  have  formally  established  performance  requirements, 
nor  threshold  values  that  can  be  used  to  evaluate  performance.  There  is  not  a  clear 
understanding  of  how  to  manage  or  assess  C4I  SoS  performance  or  C4I  SoS  capability  to 
support  Joint  or  single  Service  missions.  A  C4I  SoS-level  capability  is  a  task  achievable 
by  multiple  enterprise  components  that  is  not  achievable  by  a  single  enterprise 
component  working  on  its  own.  Examples  of  C4I  SoS  capabilities  include  Call  for  Fire, 
Immediate  Close  Air  Support,  and  Building  a  Common  Operational  Picture.  Processes 
and  methods  for  designing  and  executing  C4I  system  tests  are  well  defined  and  executed, 
but  testing  at  the  SoS  level  is  not  well  defined,  nor  are  consistent  standards  and  practices 
applied.  A  complicating  factor  is  that  real  instances  of  the  C4I  SoS  have  a  practically 
infinite  number  of  possible  configurations. 

The  Joint  Interoperability  Test  Command  (JITC)  tests  the  interoperability  of  systems,  but 
this  proves  that  system  interfaces  function.  There  is  no  agency  that  assesses  the 
capability  of  a  SoS  to  accomplish  a  task  that  requires  the  coordinated,  successful 
integration  of  functions  and  interfaces  across  multiple  systems.  The  Marine  Corps,  for 
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example,  has  extensive  doctrine  for  the  conduct  of  artillery  fire  support,  but  there  are  not 
documented,  testable  values  which  can  be  used  to  assess  the  success  of  a  fire  mission.  If 
a  Forward  Observer  (FO)  had  to  initiate  a  Call  For  Fire  five  times,  did  the  C4I  SoS 
demonstrate  a  successful  capability  to  provide  fire  support?  What  if  the  FO  had  to  try 
seven  times?  Three  times?  What  constitutes  success?  This  lack  of  a  consistent  SoS 
performance  requirement  process  bedevils  all  of  DoD. 

JC3M  system  is  important  because  it  provides  the  acquisition  community  much-needed 
performance  requirements  for  the  design  of  new  systems,  integration  of  legacy  systems, 
and  SoS  testing.  JC3M  system  is  also  important  because  it  provides  system  integrators  a 
tool  to  assess  integration  formally,  it  documents  system  capabilities  and  construction,  and 
it  provides  confidence  to  the  warfighter  that  the  C4I  SoS  works.  Every  C4I  SoS  has  been 
custom  built  to  date,  with  all  the  Configuration  Management  (CM),  troubleshooting, 
training,  and  support  challenges  this  “one-off’  approach  implies.  With  a  consistent 
assessment  methodology  and  a  documented  baseline  configuration,  C4I  SoS  support 
becomes  repeatable,  and  CM,  training,  troubleshooting,  and  knowledge  management 
costs  shrink. 

The  Joint  Test  Evaluation  Methodology  (JTEM)  team  is  addressing  Joint  SoS 
interoperability  from  the  Office  of  Secretary  of  Defense  (OSD)  level.  Marine  Corps 
Systems  Command  (MARCORSYSCOM),  the  acquisition  organization  for  the  Marine 
Corps,  is  approaching  the  issue  from  a  Service  perspective.  MARCORSYSCOM  has 
tasked  MCTSSA  to  develop  Marine  Air  Ground  Task  Force  (MAGTF)  C4I  Capability 
Certification  Management  (MC3T),  a  methodology  for  managing  the  MAGTF  C4I  SoS 
as  a  single  system,  in  accord  with  modem  systems  engineering  practices.  MC3T  will 
manage  the  MAGTF  C4I  SoS  as  a  set  of  SoS-level  capabilities,  rather  than  as  a  fixed 
hardware  or  software  baseline. 

The  JC3M  project  team  will  define  a  system  that  will  assist  a  test  agency  in  performing  a 
C4I  SoS  assessment  that  will  identify  the  desired  war  fighting  capabilities  and  ensure  that 
the  system  under  test  meets  these  requirements.  The  JC3M  system  will  include  Doctrine, 
Organization,  Training,  Materiel,  Leadership,  Personnel,  and  Facilities  (DOTMLPF) 
considerations  in  designing  the  system  to  be  used  by  an  organization,  following 
repeatable  processes,  and  consistent  resources.  The  processes  will  not  only  be  usable  by 
MC3T,  but  can,  with  appropriate  Service-specific  modifications,  be  utilized  by  other 
Services  and  Joint  agencies. 

ORGANIZATION 

The  JC3M  project  team  organization  is  provided  in  Figure  45. 

JC3M 

POINT  OF  CONTACT 
\ _ _ _ / 


EDITOR 
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Figure  45.  JC3M  Team  Organization. 
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Roles  &  Responsibilities 

The  JC3M  project  team  consists  of  14  interdisciplinary  team  members  located  in  5 
different  geographical  locations.  Table  19  is  a  listing  of  each  team  member’s  roles  and 
responsibilities.  Roles  and  responsibilities  are  subject  to  change  as  the  project  progresses. 


Name 

Roles  &  Responsibilities 

Location 

Lamb,  Jeremy 

Librarian  &  Researcher 

Bethesda 

Martin,  Calvin 

Meeting  Minutes,  Librarian  &  Researcher 

China  Lake 

Acosta,  Jacobo 

Business  &Cost 

Corona 

Huseth,  Scott 

Meeting  Minutes,  Business  &  Cost 

Corona 

Medina,  Vince 

Business  &  Cost,  Modeling  &  Simulation 

Corona 

Trinh,  Khai 

Modeling  &  Simulation 

Corona 

Hoesly,  Scott 

Meeting  Minutes,  Business  &  Cost 

MCTSSA 

Medina,  Jorge 

Business  &  Cost 

MCTSSA 

Nguyen,  Michael 

Librarian  &  Researcher 

MCTSSA 

Patel,  Jay 

Project  Schedule,  Librarian  &  Researcher 

MCTSSA 

Rangi,  Kamaljit 

Librarian  &  Researcher,  Modeling  &  Simulation,  Project  Schedule 

MCTSSA 

Schoen,  Tim 

Librarian  &  Researcher,  Modeling  &  Simulation 

MCTSSA 

Willis,  Ron 

POC,  Business  &  Cost 

MCTSSA 

Krider,  Steven 

Editor  in  Chief 

Philadelphia 

Table  19.  Team  Member  Listing. 

Project  Advisors 

The  JC3M  project  team  advisor  is  Gregory  Miller.  The  second  reader  is  Professor  David 
Hart.  Both  are  NPS  faculty  members  in  the  Department  of  Systems  Engineering. 

APPROACH  FOR  PMP  UPDATES 

The  JC3M  project  team  will  review  the  PMP  throughout  the  NPS  Capstone  project.  If 
significant  discrepancies  or  errors  are  found  during  a  review,  the  PMP  will  be  updated. 
The  Editor  in  Chief  will  incorporate  the  changes  and  submit  the  revised  PMP  to  the 
Capstone  Advisor  for  review  and  approval.  If  a  change  in  scope  or  engineering  approach 
induced  the  revision,  the  JC3M  project  team  will  resubmit  the  PMP  to  the  Systems 
Engineering  Department  Chair  for  approval. 

CONFIGURATION  MANAGEMENT 

The  deliverables  created  by  the  JC3M  project  are  in  the  form  of  documents, 
presentations,  and  simulation  results.  The  Editor  in  Chief  will  maintain  a  copy  of  all 
deliverables  and  deliverable  revisions  in  a  chronological  archive.  The  JC3M  project  team 
will  revise  each  of  the  deliverables  prior  to  being  finalized  for  submittal.  This  JC3M 
project  team  edits  will  be  tracked  utilizing  a  system  of  incremental  alphanumeric 
revisions  (i.e.  PMP  Rev  A  to  PMP  Rev  B).  Once  a  deliverable  is  ready  for  submittal  it 
will  be  published  utilizing  numeric  revision  (i.e.  PMP  Rev  1). 

TECHNICAL  REVIEWS 
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The  JC3M  project  team  will  employ  technical  reviews  as  a  way  to  ensure  accuracy, 
validity,  and  appropriateness  of  progress.  Work  products  will  be  reviewed  internally  and 
by  stakeholders  to  ensure  all  parties  understand  the  problem,  the  approach,  and  the 
solution.  Technical  reviews  precede  formal  control  gates,  set  expectations  for  all 
stakeholders,  and  reduce  the  number  of  surprises  at  formal  reviews. 


SYSTEMS  ENGINEERING  APPROACH 


OVERVIEW 


The  JC3M  project  team  will  implement  a  Systems  Engineering  approach  that  starts  with 
the  identification  of  customers’  needs  and  proceeds  through  the  phases  illustrated  in 
Figure  46  until  a  recommended  solution  is  generated.  Each  phase  presents  the 
opportunity  to  re-evaluate  progress  and  return  to  a  prior  stage  for  refinements  if 
necessary.  This  will  be  critical  to  adapting  the  project  to  ensure  the  customers’  and 
stakeholders’  needs  are  being  achieved.  The  SE  process  model  below  is  based  on 
combination  of  System  Engineering  Design  Process  (SEDP)  by  Prof  Paulo  and 
SIMILAR  System  Engineering  Process  Model  from  INCOSE,  modified  to  fit  this  project. 


Figure  46.  JC3M  Systems  Engineering  Process. 

JC3M  Systems  Engineering  Process  Phases 

Each  of  the  JC3M  Systems  Engineering  Process  phases  is  explained  herein. 

Problem  Refinement  Phase 

The  JC3M  project  team  will  utilize  this  phase  to  clarify  the  customer’s  needs  and  begin 
managing  the  JC3M  project  risks.  The  outputs  of  this  phase  will  allow  the  JC3M  project 
team  to  design  multiple  solutions  for  the  JC3M  system  problem.  The  Problem 
Refinement  Phase  is  composed  of  four  key  elements  described  below. 
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Element  1 :  Needs  Analysis 

This  element  will  provide  justification  for  proceeding  further  in  the  design  process.  It 
will  perform  System  Decomposition  in  order  to  gain  an  initial  understanding  of  the 
process  in  question  and  to  begin  to  look  at  the  process  in  terms  of  how  it  fits  into  the  “big 
picture”,  and  Functional  Analysis  in  order  to  identify  and  decompose  the  critical 
function(s).  It  will  take  primitive  needs  elicited  during  stakeholder  analysis  and 
transforming  them  into  effective  needs.  If  conflicts  occur,  compromise  will  be  sought 
after  when  possible,  however,  key  stakeholders  such  as  JTEM  and  the  NPS  Systems 
Engineering  Department  will  have  stronger  influence  than  others. 

Inputs:  Initial  Problem  Statement,  Stakeholders  Needs/Wants 

Tools:  Functional  flow  diagrams 

Outputs:  Revised  Problem  Statement 

Element  2:  Requirement  Generation  and  Analysis 

During  this  element,  the  JC3M  project  team  will  generate  a  set  of  processes  that  reflect 
stakeholders’  needs.  Requirements  analysis  will  assists  the  customers  in  refining  their 
requirements  in  concert  with  defining  functional  requirements.  Stakeholder  Analysis  will 
also  be  performed  to  identify  people  and  organizations  relevant  to  our  problem  and  to 
determine  their  needs,  wants  and  desires.  Stakeholders  will  have  a  vested  interest,  or 
personal  stake,  in  our  problem  and/or  its  eventual  solution. 

Inputs:  Revised  Problems  Statement 

Outputs:  Initial  Problem  Refinement  Report  (PRR),  Functional  Hierarchy 
Element  3:  Value  System  Design 

The  JC3M  project  team  will  use  this  element  to  help  establish  processes,  objectives,  and 
evaluation  measures.  Value  System  Design  will  establish  a  qualitative  value  hierarchy 
tree  that  identifies  processes,  objectives  and  sub-objectives,  and  evaluation  criteria. 

Value  System  Design  will  identify  system  characteristics  that  reveal  measurable 
parameters.  These  parameters  will  identify  the  metrics  for  evaluation  of  alternative 
solutions.  Measures  of  Effectiveness  (MOE)  and  Measures  of  Performance  (MOP)  will 
be  generated  from  the  value  hierarchy  tree.  MOEs  will  reflect  operational  objectives  and 
be  understandable  to  decision-makers.  MOPs  will  provide  measurable  results  that  can 
demonstrate  progress  towards  goals  and  objectives,  provide  specific  measurement  results, 
and  determine  effectiveness.  MOPs  will  determine  if  the  system  is  meeting  its  mission, 
vision,  and  goals. 

Inputs:  Initial  PRR 

Tools:  Value  Hierarchy  Tree 

Outputs:  MOEs,  MOPs 
Element  4:  Initiate  Risk  Management 

The  JC3M  project  team  shall  Initiate  Risk  Management  by  identifying  potential 
opportunities  for  risks,  assessing  their  associated  probabilities  of  occurrence,  and 
determining  their  impact  to  the  project.  Risk  management  will  address: 

Requirements  risk  -  The  JC3M  project  team  will  identify  and  describe  the  requirements 
of  JC3M  system  with  stakeholders,  and  aggressively  manage  the  scope  of  requirements  to 
minimize  scope  creep. 

External  risks  -  Is  the  process  development  dependent  on  external  events  or 
accomplishments  over  which  the  program  has  no  control? 
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Development  risk  -  Can  the  process  be  designed  so  that  required  function  and 
performance  are  met  within  constraints? 

Resource  availability  -  Are  required  personnel  and  facilities  available? 

Resource  risks  -  Are  funding,  training,  schedule  (time)  and  tools  available? 

The  JC3M  project  team  will  manage  these  risks  by  developing  and  documenting  a  risk 
management  strategy.  This  strategy  will  include  risk  planning,  assessment,  handling,  and 
monitoring.  For  each  identified  risk  item,  probability/consequence  scales  will  be 
developed  and  ratings  will  be  assigned.  Supporting  analysis  will  help  rate,  prioritize,  and 
aggregate  risks  in  order  to  minimize  potential  areas  of  concern.  This  analysis  will  be 
conducted  in  accordance  with  the  Risk  Management  Guide  For  DoD  Acquisition. 

Design  Alternatives  Phase 

The  JC3M  project  team  will  utilize  the  Design  Alternatives  Phase  to  generate  multiple 
solutions  to  the  problem,  establish  feasibility  criteria  and  apply  those  criteria  to  eliminate 
those  alternatives  that  are  clearly  infeasible.  The  creation  of  these  solutions  requires  the 
outputs  of  the  Problem  Refinement  Phase.  The  outputs  of  this  phase  will  be  used  for  the 
creation  of  models  in  the  Model  the  Alternatives  Phase  and  for  the  Analysis  of 
Alternative  (AoA)  portion  of  the  Simulation  and  Analysis  of  Alternatives  Phase.  The 
Design  Alternatives  Phase  is  composed  of  three  key  elements. 

Element  1 :  Alternative  Generation 

In  this  element  the  JC3M  project  team  will  prioritize  the  critical  functions  and  sub¬ 
functions  that  the  system  under  design  must  be  able  to  perform.  These  critical  functions 
will  be  a  subset  of  the  functions  defined  in  the  Value  Hierarchy  during  Value  Systems 
Design.  Next,  the  JC3M  project  team  will  brainstorm  and  research  alternative  ways  to 
perform  the  critical  system  functions.  Finally,  the  JC3M  project  team  will  build 
alternative  packages  that  account  for  each  function  identified.  Zwicky’s  Morphological 
Box  (ZMB)  will  be  used  as  a  tool  to  develop  the  alternative  packages.  The  output  of  this 
element  will  be  a  set  of  alternative  solutions  for  the  system  under  development. 

Inputs:  Value  Hierarchy  Tree,  MOEs,  MOPs,  and  PRR 

Tools:  ZMB,  brainstorming,  and  research 

Outputs:  Set  of  alternatives 

Element  2:  Feasibility  Screening 

The  purpose  of  this  element  is  to  eliminate  from  further  consideration  those  alternatives 
that  are  clearly  infeasible  so  as  not  to  waste  valuable  effort  in  the  Model  the  Alternatives 
Phase.  First,  screening  constraints  will  be  defined.  Most  of  the  screening  constraints  will 
come  directly  from  the  constraints  on  the  system  detailed  in  the  Value  System  Design 
element  of  the  Problem  Refinement  Phase.  The  JC3M  project  team  will  also  obtain 
screening  constraints  from  the  stakeholders. 

Those  alternatives  that  meet  all  of  the  system  constraints  are  considered  feasible,  while 
those  that  clearly  fail  to  meet  one  or  more  constraints  are  considered  infeasible.  The 
screening  process  will  only  screen  out  those  alternatives  that  are  clearly  infeasible.  The 
JC3M  project  team  will  use  a  Feasibility  Screening  Matrix  (FSM)  to  organize  the  results 
and  identify  the  feasible  alternatives. 

Inputs:  Set  of  alternatives,  SRD,  MOEs,  MOPs,  and  Stakeholder  input 

Tools:  FSM 

Outputs:  Feasible  alternatives 
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Element  3:  Alternative  Scoring  Criteria 

The  purpose  of  this  element  is  to  define  the  criteria  for  scoring  alternatives  based  on 
MOEs  and  MOPs.  Stakeholders  must  agree  on  the  values  that  will  be  later  used  to  score 
the  final  alternatives.  Numbers  quantify  values  and  uncertainties;  humans  can 
understand,  compare,  and  manipulate  numbers.  These  criteria  will  be  generated  early,  so 
stakeholders  can  review  and  validate  them.  Alternative  scoring  criteria  provide 
quantitative  support  for  the  decision-makers  and  enables  consistent  evaluation  of 
alternatives.  The  model  that  we  will  use  is  the  quantitative  value  model.  Quantitative 
value  models  are  the  scoring  functions  and  weights  that  will  be  used  during  the  Analysis 
of  Alternatives  to  evaluate  and  compare  our  alternatives. 

Input:  Feasible  alternatives,  MOEs,  MOPs,  Value  Hierarchy  Tree 

Output:  Scoring  Criteria,  Preliminary  Quantitative  Value  Modeling  Decision  Matrix 

Model  the  Alternatives  Phase 

The  JC3M  project  team  will  utilize  this  phase  to  generate  models  based  on  the 
alternatives  selected  in  the  Design  Alternatives  Phase.  The  outputs  of  this  phase  are  the 
models  of  the  alternatives  that  will  be  simulated  during  the  Simulation  and  Analysis  of 
Alternatives  Phase.  The  Model  the  Alternatives  Phase  is  composed  of  the  two  key 
elements  described  below. 

Element  1 :  Model  Development 

Models  will  be  developed  based  on  the  alternatives  selected  in  the  Design  Alternatives 
phase  in  accordance  with  Department  of  Defense  Architecture  Framework  (DoDAF), 
MAGTF,  and  Joint  Services  architectures.  SoS  interoperability  diagrams  based  on  legacy, 
current,  and  future  systems  will  be  utilized  to  create  detailed  functional  and  behavioral 
models. 

Inputs:  Feasible  alternatives,  DoDAF,  Joint  Service  architectures,  and  Functional 

Hierarchy 

Tools:  Microsoft  Visio  and  Vitech  COREsim 

Outputs:  Detailed  functional  and  behavioral  models 
Element  2:  Cost  Analysis 

Based  on  the  system  requirements,  the  system  life  cycle,  and  activities  in  each  phase  of 
the  life  cycle,  the  Cost  Breakdown  Structure  (CBS)  will  be  developed  and  a  life  cycle 
cost  estimate  for  each  alternative  will  be  developed.  This  will  include  a  business  model 
for  the  activity  performing  the  test  functions  detailed  in  the  final  process. 

Inputs:  Detailed  functional  and  behavioral  models,  existing  cost  data  (stakeholders 

and  others  will  be  consulted  for  existing  cost  data) 

Tools:  Microsoft  Excel  and  Operating  and  Support  Cost  Analysis  Model  (OSCAM) 

Outputs:  SoS  Test  Process  Cost  Estimates 

Simulation  and  Analysis  of  Alternatives  Phase 

The  JC3M  project  team  will  utilize  the  models  generated  earlier  to  evaluate  and  rank  each 
of  the  alternatives.  The  output  of  this  phase  will  provide  a  recommended  solution  to  be 
published  in  the  Final  Recommendation  Phase.  The  Simulation  and  Analysis  of 
Alternatives  Phase  is  composed  of  the  two  key  elements  described  below. 

Element  1:  Simulation 
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The  simulation  model  will  be  probabilistic,  with  elements  of  uncertainty  as  defined  in  the 
Problem  Refinement  Phase.  The  results  from  the  simulation  will  provide  a  means  of 
predicting  or  estimating  the  performance  of  our  design  alternatives  with  respect  to  the 
selected  evaluation  measures. 

Input:  Detailed  functional  and  behavioral  models 

Tools:  Vitech  COREsim ,  SimVision  and  Arena 

Outputs:  Simulation  results 
Element  2:  Sensitivity  Analysis 

The  JC3M  project  team  will  utilize  sensitivity  analysis  to  determine  the  factors  that 
contribute  the  most  to  variability  in  the  simulation  outputs. 

Input:  Simulation  results 

Outputs:  Factors  that  contribute  to  the  output  variability 
Element  3:  Trade  off  Studies 

Trade-off  studies  for  the  models  and  will  be  based  on  cost,  schedule,  and  performance  of 
each  alternative. 

Input:  Simulation  results 

Outputs:  Preliminary  alternative  ranking 

Element  4:  Analysis  of  Alternatives 

This  phase  compares  the  alternative  models,  after  their  simulation  performance,  using 
MOEs,  MOPs,  and  Quantitative  Value  Modeling  Decision  Matrix,  to  rank  the 
alternatives.  Once  the  structure  and  numbers  are  in  place,  and  the  stakeholders  agree,  the 
analysis  can  begin. 

Inputs:  Simulation  results,  factors  that  contribute  to  the  output  variability,  scoring 

criteria,  Preliminary  Quantitative  Value  Modeling  Decision  Matrix,  Preliminary 
alternative  ranking 

Tools:  Quantitative  Value  Modeling  Decision  Matrix 

Outputs:  Ranked  Alternatives 

Final  Recommendation  Phase 

The  JC3M  project  team  will  assemble  all  earlier  inputs  into  a  cohesive  recommendation 
for  implementation,  and  publish  this  as  a  final  report.  The  output  of  the  Final 
Recommendation  Phase  will  be  provided  to  both  the  ITEM  and  MC3T  teams  for  their 
use.  The  MC3T  implementation  team  is  a  critical  stakeholder,  but  their  schedule  includes 
a  proof  of  concept  event  in  October  2007.  Based  on  this  schedule,  the  MC3T  team 
concluded  they  cannot  wait  for  JC3M  project  outputs.  MC3T  will  instead  implement  an 
interim  process,  and  will  review  the  JC3M  project  report  for  inclusion  of  applicable 
recommendations  as  MC3T  refines  their  processes.  The  Final  Recommendation  Phase  is 
composed  of  one  key  element  described  below. 

Element  1 :  Publish  Recommendation 

In  this  phase,  all  outputs  from  earlier  phases  will  be  assembled  into  a  final  report.  This 
report  will  be  briefed  and  provided  to  stakeholders  at  the  end  of  the  project. 

Inputs:  Ranked  Alternatives 

Outputs:  Final  Report  describing  a  recommended  approach  for  the  conduct  of  C4I  SoS 
capability  testing 
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Assess  Performance  Phase 

This  project  will  not  completely  conclude  after  delivery  of  the  final  report.  Some 
members  of  the  JC3M  project  team  will  continue  to  work  with  MC3T,  ITEM,  and  other 
stakeholders  after  this  project  concludes.  Some  members  of  the  team  already  participate 
in  the  JTEM  Capability  Testing  Community  of  Interest,  where  they  will  share  their 
experiences  with  other  members  and  continue  to  advance  the  art  of  capability  testing. 

STAKEHOLDERS 

Identification  of  stakeholders  -  organizations  and  personnel  with  an  interest  in,  and  some 
authority  over,  the  final  content  of  the  project  -  is  a  critical  task.  Too  few  stakeholders 
can  result  in  an  incomplete  problem  assessment,  with  a  resulting  solution  that  is  narrow. 
Too  many  stakeholders,  with  varying  interests,  can  dilute  the  focus  of  the  project,  or 
increase  the  scope  beyond  the  ability  of  the  project  team  to  address  within  a  limited  time. 
The  project  team  will  review  the  project  history  and  description  to  identify  stakeholders. 
The  team  will  interview  the  stakeholders  to  validate  their  interest  and  authority,  as  well  as 
to  identify  possible  additional  stakeholders. 

The  stakeholders  are: 

•  NPS  Systems  Engineering  Department  will  validate  the  appropriate  problem  and 
solution  approach. 

•  JTEM  team,  Joint  Test  and  Evaluation  Project,  is  tasked  to  develop,  test,  and 
validate  a  methodology  for  defining,  developing,  and  using  a  test  environment  to 
support  test  and  evaluation  of  system  performance  in  a  Joint  mission  operational 
environment.  JTEM  reports  through  the  Deputy  Director,  Air  Warfare 
Operational  Test  and  Evaluation,  to  Director,  Operational  Test  and  Evaluation 
(DOT&E)  to  the  OSD.  The  JTEM  lead  is  very  interested  in  how  both  the  JC3M 
project  and  MC3T  solve  some  of  the  same  challenges  JTEM  faces. 

•  MC3T  team  at  MCTSSA,  which  may  use  the  project  output  to  modify  their  C4I 
SoS  testing  processes. 

•  Marine  Corps  Combat  Development  Command  (MCCDC)  is  the  agency  that 
defines  doctrine  and  requirements  for  the  Marine  Corps.  MCCDC  will  be 
engaged  in  defining  MAGTF  C4I  SoS  requirements,  and  will  be  included  in 
MC3T  processes. 

Possible  additional  stakeholders: 
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•  Central  Technical  Support  Facility  (CTSF),  Ft.  Hood,  TX,  which  performs  many 
SoS  tests  for  the  U.S.  Army,  like  MCTSSA,  and  may  be  interested  in  SoS  test 
requirements  generation  and  conduct. 

•  U.S.  Army  Test  and  Evaluation  Command  (ATEC),  Alexandria,  VA,  plans, 
conducts,  and  integrates  developmental  testing,  independent  operational  testing, 
independent  evaluations,  assessments,  and  experiments. 

•  U.S.  Navy  Operational  Test  and  Evaluation  Force  (OPTEVFOR),  Norfolk,  VA, 
assesses  the  operational  effectiveness  and  suitability  of  new  and  improved  war 
fighting  systems  and  capabilities  for  the  Navy. 

•  JITC  has  the  mission  to  provide  operational  test  and  evaluation  of  C4I  systems, 
certify  C4I  systems  interoperability  and  solve  interoperability  deficiencies. 

•  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base,  NM, 
assesses  the  capability  of  new  systems  to  meet  warfighter  needs  by  planning, 
executing  and  reporting  independent  operational  evaluations. 

REQUIREMENTS 

At  this  stage  of  the  project,  the  JC3M  project  team  knows  that  the  system  to  be  designed 
must  be  relatively  affordable,  flexible  enough  to  work  with  a  variety  of  SoS 
configurations,  and  relatively  quick  to  execute.  As  defined  in  the  process  approach, 
refinement  of  requirements,  as  well  as  the  possible  discovery  of  new  requirements,  may 
affect  a  balanced,  life-cycle  solution.  Additional  requirements  will  be  identified  during 
the  needs  analysis  element  of  the  Problem  Refinement  Phase  by  conducting  system 
decomposition,  stakeholder  analysis,  and  functional  analysis,  These  requirements  will  be 
documented  in  the  SRD  that  will  be  reviewed  and  approved  by  the  stakeholders  prior  to 
the  Design  Alternatives  phase. 

TOOLS 

The  JC3M  project  team  plans  to  utilize  the  tools  identified  in  each  phase  and  element 
during  the  execution  of  the  project.  However,  the  JC3M  project  team  may  determine  that 
the  identified  tools  are  inadequate,  unnecessary,  or  undesirable.  If  this  occurs  new  tools 
will  be  researched  and  selected,  if  required,  to  complete  the  project  elements. 

RISK  MANAGEMENT 

Risks  to  the  project  will  be  defined  and  managed  by  the  business  and  cost  sub-team. 

Risks  will  be  identified  during  project  analysis,  confirmed  with  primary  stakeholders,  and 
continually  managed.  Risks  will  be  managed  by  prioritization  and  recording,  creation  of 
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a  risk  management  plan  for  each  significant  risk,  and  ongoing  reporting  of  risks  and 
associated  issues.  Initial  risks  are  identified  below,  along  with  their  mitigation  approach. 
Personnel  risk  is  medium.  Student  teams  will  be  assigned  to  each  task.  This  project 
should  be  the  focus  of  the  student’s  efforts,  which  reduces  the  risk  of  personnel 
reassignment.  This  risk  will  be  reduced  by  ongoing  communication  between  project  sub¬ 
teams.  Personnel  outside  the  student  teams  (references,  authorities,  and  contributors) 
may  be  reassigned,  unavailable,  or  slow  to  respond,  and  thus  present  a  medium  risk.  This 
risk  will  be  managed  by  ongoing  communication  with  outside  personnel,  as  well  as 
communications  with  stakeholders  on  the  progress  of  the  project,  or  schedule  slips. 

Time  risk  is  medium.  The  project  must  be  completed  by  September  2007,  which  strictly 
limits  this  resource.  If  the  project  grows  from  the  current  scope,  there  may  not  be 
sufficient  time  to  complete  the  project.  This  risk  will  be  managed  by  aggressively 
monitoring  the  scope  of  the  project,  communicating  with  stakeholders,  and  managing 
scheduled  activities. 

Lack  of  resources  is  a  low-medium  risk.  The  primary  equipment  need  identified  to  date 
is  a  simulation  tool  for  modeling  processes.  The  use  of  these  tools  introduces  some  risk 
(availability,  speed  to  learn,  suitability),  but  this  will  be  managed  by  utilizing  simple, 
accessible,  and  suitable  tools  where  appropriate.  Because  this  is  a  student-run  (unfunded) 
project,  there  is  a  medium  risk  of  not  having  suitable  simulation  tools  due  to  lack  of 
funds.  If  funds  are  lacking,  no-  or  low-cost,  non-simulation-specific  tools  will  be 
identified  for  use. 
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MILESTONES  &  DELIVERABLES 

Table  20  lists  each  milestone  and  Interim  Progress  Review  (IPR)  associated  with  this 
project.  Each  milestone  will  have  at  least  one  scheduled  meeting  with  the  stakeholders  to 


discuss  decisions  and  deliverables  for  that  milestone.  The  stakeholders  and  the  JC3M 
project  team  must  agree  that  the  required  deliverable(s)  are  completed  satisfactorily 
before  a  milestone  is  considered  complete.  _ _ 


Milestone 

IPR 

Description 

Deliverable 

Date 

1 

- 

PMP  Approval 

Project  Management  Plan 

16  Feb  2007 

2 

1 

Requirements 

Approval 

System  Requirements 
Document 

16  Mar  2007 

3 

2 

Alternative  Scoring 
Matrix  Approval 

Alternative  Matrix 

18  Apr  2007 

4 

3 

Models  of  Alternatives 

Conceptual  Alternative 
Models  &  Descriptions 

21  Jun  2007 

5 

4 

Alternative  Selection 

Alternative  Selection 
Report 

16  Aug  2007 

6 

5 

Report  Delivery  & 
Project  Presentation 

Project  Presentation  and 
Final  Report 

14  Sep  2007 

Table  20.  List  of  Milestones  and  Deliverables. 
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SCHEDULE 


The  schedule  for  the  JC3M  project  team  is  provided  in  Figure  47 
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APPENDIX  B.  NEEDS  ANALYSIS  QUESTIONNAIRE 


After  the  definition  of  the  primitive  need,  requirements  were  captured  via 
stakeholder  focused  interviews,  aided  by  questionnaires.  The  multiple  responses  to  the 
first  questionnaire  were  received  with  mixed  results.  All  of  the  responses  have  not  been 
incorporated  due  to  lack  of  quality  and  importance.  The  questionnaires  were  reviewed 
verbally  with  the  stakeholder  or  provided  to  them  for  review  electronically;  responses 
from  individuals  and  organizations  follow  the  questionnaire.  After  reviewing  the 
responses  from  the  first  questionnaire  the  team  decided  to  adapt  their  approach  and 
generate  a  second  questionnaire  with  more  pertinent  questions.  JITC’s  responses  from  the 
second  questionnaire  are  provided. 

QUESTIONNAIRE  VERSION  1. 

SYSTEM  OF  SYSTEM  PERFORMANCE  ASSESSMENT 

As  a  Systems  Engineering  project  for  the  Naval  Postgraduate  School,  Monterey, 
CA,  students  are  creating  a  system  for  certifying  that  a  Command,  Control, 
Communications,  Computers,  and  Intelligence  (C4I)  System  of  Systems  (SoS)  can 
provide  stated  capabilities.  Students  are  contacting  organizations  in  order  to  understand 
current  practices,  establish  a  performance  baseline,  and  identify  alternative  approaches  to 
a  solution. 

Your  organization,  which  designs,  creates,  or  maintains  a  complex  SoS,  has  been 
identified  as  a  candidate  to  answer  this  questionnaire  and  participate  in  focused 
interviews,  An  example  SoS  would  be  a  commercial  passenger  aircraft  developed  as  a 
single  system,  which  incorporates  components  developed  for  other  aircraft,  rather  than 
developing  new  components.  The  SoS  would  be  the  airframe,  avionics,  engines,  radar, 
etc.  that  make  up  the  entire  aircraft  capability.  Your  support  is  requested  because  it  will 
directly  assist  the  student  team,  advance  the  art  of  Systems  Engineering,  and  increase  the 
efficiency  and  effectiveness  of  C4I  SoS  testing. 

1.  SoS  Definition: 

a.  What  is  your  SoS? 

b.  Where  is  it  used? 

c.  What  does  it  do? 

d.  What  are  the  high-level  components  of  your  SoS? 
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e.  Are  components  created  or  managed  independently,  or  are  they  managed 
centrally? 

f.  Was  your  SoS  built  to  a  systems  architecture  model,  as  an  automobile 
might  be,  or  did  it  evolve,  as  the  Internet  has? 

2.  Requirements: 

a.  Who  creates  requirements  for  your  SoS? 

b.  Are  requirements  generated  within  or  outside  your  organization? 

c.  Does  the  requirements  generator  have  any  authority  over  construction, 
operation,  or  maintenance  of  the  SoS? 

d.  Do  requirements  for  the  SoS  change  over  time? 

e.  Are  requirements  documented? 

f.  Can  I  review  your  requirements? 

3.  Validation  and  Verification  (V&V): 

a.  Is  validation  (“does  it  do  the  right  things?”)  or  verification  (“does  it 
perform  to  specifications?”)  required  for  your  SoS,  or  for  component 
systems? 

b.  Who  performs  V&V  for  your  SoS?  Is  this  an  internal  or  external 
organization? 

c.  What  are  sources  of  V&V  requirements:  are  they  derived  from  system 
requirements,  user  requirements,  or  other  sources? 

d.  What  are  the  consequences  of  V&V  failure? 

e.  How  is  V&V  performed  for  your  SoS? 

f.  If  components  of  the  SoS  change,  is  testing  performed? 

g.  Can  I  view  your  test  procedures  and  results  reports? 

h.  Is  there  a  process  to  define: 

i.  test  environment, 

ii.  test  criteria, 

iii.  test  procedures, 

iv.  test  conduct,  and 

v.  results  reporting? 

4.  Support: 

a.  Is  your  SoS  supported  by  an  internal  or  external  organization? 

b.  How  is  your  SoS  maintained? 

c.  Is  maintenance  performed  by  an  internal  or  external  organization? 

d.  Is  the  SoS  subject  to  change? 

e.  How  is  the  configuration  of  the  SoS  initially  recorded,  how  is  it  checked, 
and  how  is  it  controlled? 

f.  If  part  of  the  SoS  is  inoperative,  how  do  users  request  service? 

g.  How  are  users  trained  on  functions  of  the  SoS? 

h.  Are  there  user,  support,  or  other  training  resources  I  can  view? 

i.  How  are  support  personnel  trained? 
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5.  SoS  Operations: 

a.  What  is  the  life  cycle  of  your  SoS? 

b.  Does  your  SoS  operate  continuously  or  intermittently? 

c.  How  long  do  you  expect  your  SoS  to  function? 

d.  Does  your  SoS  have  documentation? 

e.  Can  I  view  the  documentation? 

f.  Is  there  an  alternative  or  backup  to  your  SoS? 

g.  What  are  the  consequences  (risk  of  injury  or  loss  of  life,  risk  of  damage  or 
loss  of  equipment,  risk  of  financial  loss,  other  risk)  of  SoS  failure? 

6.  Your  Organization: 

a.  How  long  has  your  organization  been  in  operation? 

b.  Does  your  organization  follow  INCOSE,  IEEE,  ANSI  or  other  process 
standards? 

c.  How  long  have  you  operated  (sold,  created,  supported)  your  SoS? 

d.  Is  there  anything  you  would  like  to  add  about  your  organization  or  SoS? 


MC3T  Response  to  Questionnaire  Version  1 

1.  SoS  Definition: 

a.  What  is  your  SoS?  Marine  Corps  Systems  Command 

( MARCORSYSCOM)  is  developing  a  process  to  manage  the  certification 
of  the  capability  of  Command,  Control,  Communications,  Computers,  and 
Intelligence  (C4I)  Systems  of  Systems,  used  by  a  Marine  Air  Ground  Task 
Force,  to  perform  their  required  functions.  The  process  is  called  MAGTF 
C4I  Capability  Certification  Test  -  MC3T. 

b.  Where  is  it  used?  MC3T  will  be  a  distributed  operations,  used  at  Marine 
Corps  Tactical  Systems  Support  Activity  (MCTSSA),  Camp  Pendleton,  CA; 
Marine  Corps  Systems  Command,  Quantico,  VA;  and  other  sites. 

MCTSSA  performs  C4I  systems  interoperability  testing  and  support,  as 
tasked  by  the  Deputy  Commander  for  C4I  Integration  (C4I/I)  at 
MARCORSYSCOM. 

c.  What  does  it  do?  MC3T  will  assist  in  managing  the  MAGTF  C4I  SoS  as  a 
single  system,  in  terms  of  enterprise  capabilities.  With  that  definition, 
MC3T  will  test  to  ensure  the  SoS  performs  to  requirements;  MC3T  will 
also  certify  for  the  Operating  Forces  that  if  the  SoS  is  configured  in  a 
defined  manner,  it  will  meet  performance  requirements.  In  the  former 
role,  MC3T assists  acquisition  managers  to  ensure  their  systems  support 
SoS  level  capabilities;  in  the  latter  role,  MC3T  assist  the  operating  forces 
use  the  SoS  effectively. 

d.  What  are  the  high-level  components  of  your  SoS?  MC3T functional 
components  consist  of  requirements  definition,  capability  definition, 
certification,  and  documentation. 
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e.  Are  components  created  or  managed  independently,  or  are  they  managed 
centrally?  MC3T functional  components  are  managed  independently. 

i.  Marine  Corps  Combat  Development  Command  (MCCDC) 
creates  doctrine  (how  components  are  used)  and  requirements 
(what  components  must  do)  for  the  Marine  Corps. 

ii.  MARCORSYSCOM  Program  Managers  acquire  components, 
while  a  separate  MARCORSYSCOM  agency  (C4I/I)  manages 
SoS  integration. 

iii.  Marine  Corps  Operational  Test  and  Evaluation  Activity 
(MCOTEA)  performs  Operational  Tests;  program  managers 
perform  development  testing. 

f.  Was  your  SoS  built  to  a  systems  architecture  model,  as  an  automobile 
might  be,  or  did  it  evolve,  as  the  Internet  has?  The  MAGTF  C4I  SoS  has 
evolved  to  its  current  state.  MC3T  will  be  built  to  a  systems  architecture. 

2.  Requirements: 

a.  Who  creates  requirements  for  your  SoS? 

i.  MCCDC  defines  the  requirements  for  systems,  but  not  the  SoS; 

ii.  MARCORSYSCOM  Program  Managers  acquire  systems  which 
must  meet  requirements; 

iii.  MARCORSYSCOM  C4I/I  determines  C4I  integration 
requirements,  and  leads  MC3T  development  with  MCTSSA 
support.  MCCDC  will  work  with  C4I/I  and  MCTSSA  to  define 
enterpriseOlevel  requirements. 

iv.  MCOTEA  defines  how  systems  are  tested,  and  conducts  or 
coordinates  operational  tests  only. 

b.  Are  requirements  generated  within  or  outside  your  organization?  See 
above. 

c.  Does  the  requirements  generator  have  any  authority  over  construction, 
operation,  or  maintenance  of  the  SoS? 

i.  No.  MCCDC  has  no  responsibility  for  the  construction,  operation, 
or  maintenance  of  the  SoS. 

ii.  MCOTEA  has  no  responsibility  for  the  construction,  operation,  or 
maintenance  of  the  SoS. 

iii.  MARCORSYSCOM  Program  Managers  have  no  responsibility  for 
the  construction,  operation,  or  maintenance  of  the  SoS. 

iv.  MARCORSYSCOM  C4I/I  has  no  responsibility  for  the  acquisition 
of  the  SoS,  but  is  responsible  for  the  integration  of  the  SoS 
components. 

d.  Do  requirements  for  the  SoS  change  over  time?  Yes 

e.  Are  requirements  documented?  SoS  requirements  are  documented  a 
component  (C4I  system)  level,  but  not  at  the  SoS  level.  One  MC3T  goal  is 
to  define  SoS  level  requirements. 

f.  Can  I  review  your  requirements?  As  requirements  are  generated  they  will 
be  made  available. 


136 


3.  Validation  and  Verification  (V&V): 

a.  Is  validation  (“does  it  do  the  right  things?”)  or  verification  (“does  it 
perform  to  specifications?”)  required  for  your  SoS,  or  for  component 
systems? 

i.  No.  The  MAGTF  C4I  SoS  exists,  and  has  not  been  required  to 
have  validation  or  verification  performed. 

ii.  Components  of  the  SoS  that  are  a  Program  of  Record  do  have 
V&  V performed,  to  ensure  they  meet  their  system  level 
requirements. 

iii.  MARCORSYSCOM  C4I/I  chartered  Federation  Of  Systems 
(FEDOS)  testing  in  the  past.  FEDOS  defined  a  limited  C4I  SoS, 
with  a  controlled  configuration  of  hardware  and  software. 

b.  Who  performs  V&V  for  your  SoS?  Is  this  an  internal  or  external 
organization? 

•  MCTSSA  conducted  FEDOS  for  MARCORSYSCOM  C4I/I. 

•  MCTSSA  will  conduct  MC 3  T  testing  for  MARCORSYSCOM 
C4I/I. 

c.  What  are  sources  of  V&V  requirements:  are  they  derived  from  system 
requirements,  user  requirements,  or  other  sources? 

•  For  the  conduct  of  FEDOS  and  predecessor  tests,  MCTSSA 
conducted  a  search  for  SoS  V&  V  requirements.  There  are  no 
formal  SoS-level  requirements  at  all.  Where  available,  V&  V 
requirements  were  determined,  in  descending  preference,  from: 

o  Integrated  Architecture  Data  Stores 
o  Doctrine 

o  Training  and  Readiness  Manuals 
o  Schoolhouse  Documents 
o  System  Manuals  &  Help  Files 
o  Unit-level  Standard  Operating  Procedures  (SOP) 
o  Subject-matter  Experts 
o  Best  guess  synthesis 

d.  What  are  the  consequences  of  V&V  failure? 

•  Prior  to  MC3T,  failure  of  V&  V,  for  a  system  in  acquisition  or 
sustainment,  was  communicated  to  the  “owning”  PM.  Because 
SoS  level  V&  V  was  not  a  requirement,  the  PM  could  ignore  the 
failure,  address  the  failure  with  a  risk  mitigation  strategy,  or 
rebuild/refine  the  system. 

•  The  consequences  of  failure  of  MC3T,  i.e.  failing  to 
demonstrate  the  defined  capability,  are  not  defined. 

e.  How  is  V&V  performed  for  your  SoS?  MC3T  is  developing  the  capability 
certification  testing  process. 

f.  If  components  of  the  SoS  change,  is  testing  performed?  MC3T  will 
incorporate  capability  testing  for  new  or  modified  components  of  the 
MAGTF  C4I  SoS. 
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g.  Can  I  view  your  test  procedures  and  results  reports?  MC3T  will  conduct 
the  first  test  event  in  October  2007.  Procedures  will  be  made  available  for 
review  as  they  are  developed.  Test  methodology  documents  are  more 
important. 

h.  Is  there  a  process  to  define: 

i.  test  environment, 

ii.  test  criteria, 

iii.  test  procedures, 

iv.  test  conduct,  and 

v.  results  reporting?  MC3T  will  conduct  the  first  test  event  in 

October  2007.  Definitions,  procedures,  and  criteria  will  be  made 

available  for  review  as  they  are  developed. 

4.  Support: 

a.  Is  your  SoS  supported  by  an  internal  or  external  organization?  The 
MAGTF  C4I  SoS  is  supported  by: 

•  MCTSSA  Operating  Forces  Tactical  Systems  Support  Center 
(OFTSSC)  provides  24x7  remote  support  for  tactical  C4I 
systems  to  the  Marine  Corps  and  other  Operating  Forces. 

•  Marine  Corps  Network  Operations  and  Security  Command 
(MCNOSC)  provides  network  defense  and  technical  support. 

•  Marine  Corps  Operating  Forces  operate  tactical  C4I  systems 
around  the  world. 

•  MARCORSYSCOM  Program  Managers  provide  varied  levels 
of  support  for  their  systems. 

b.  How  is  your  SoS  maintained?  By  the  Operating  Forces,  MCNOSC,  and 
MARCORSYSCOM. 

c.  Is  maintenance  performed  by  an  internal  or  external  organization? 
Maintenance  is  performed  by  MCNOSC  and  the  Operating  Forces;  both 
are  external  to  MCTSSA. 

d.  Is  the  SoS  subject  to  change?  Frequently. 

e.  How  is  the  configuration  of  the  SoS  initially  recorded,  how  is  it  checked, 
and  how  is  it  controlled?  This  may  be  an  operational  question,  and 
requires  vetting.  See  Annex  K  to  the  Oplan. 

f.  If  part  of  the  SoS  is  inoperative,  how  do  users  request  service?  Direct 
contact  with  MCNOSC  or  Operating  Forces  Tactical  Systems  Support 
Center. 

g.  How  are  users  trained  on  functions  of  the  SoS?  Users  may  receive 
training  on  component  systems.  There  is  no  SoS  level  training. 

h.  Are  there  user,  support,  or  other  training  resources  I  can  view?  Resources 
can  be  made  available  on  a  case-by-case  basis. 

i.  How  are  support  personnel  trained?  SoS  support  personnel  at  the  OFTSSC 
are  component  (system-level)  experts,  with  formal  training  on  a  system. 
They  are  cross  trained  on  other  systems  over  time. 
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5.  SoS  Operations: 

a.  What  is  the  life  cycle  of  your  SoS? 

•  Once  the  MAGTF  C4I  SoS  is  established,  it  remains 
operational  until  a)  the  MAGTF  is  disbanded  and  returns  to 
garrison,  or  b)  the  Marine  Corps  operation  (Combat, 
Humanitarian  Assistance,  Training  Exercise)  is  completed. 

•  MC3T  is  being  defined,  with  the  first  demonstration  of  a  test 
event  in  October  2007. 

b.  Does  your  SoS  operate  continuously  or  intermittently? 

•  The  MAGTF  C4I  SoS  operates  as  described  above. 

•  MC3T  is  not  an  SoS,  but  will  operate  on  a  continuous  basis, 
conducting  assessments  as  needed. 

c.  How  long  do  you  expect  your  SoS  to  function?  For  the  foreseeable  future. 

d.  Does  your  SoS  have  documentation?  MC3T Is  not  an  SoS,  but 
documentation  is  being  developed. 

e.  Can  I  view  the  documentation?  As  it  is  developed. 

f.  Is  there  an  alternative  or  backup  to  your  SoS?  The  current  ad-hoc  system 
is  the  alternative. 

g.  What  are  the  consequences  (risk  of  injury  or  loss  of  life,  risk  of  damage  or 
loss  of  equipment,  risk  of  financial  loss,  other  risk)  of  SoS  failure? 

•  IfMC3T  works,  it  will  reduce  systems  development, 
integration,  and  support  costs;  the  current  ad-hoc  system  costs 
more  in  all  these  areas. 

•  MC3T  will  also  increase  the  effectiveness  and  efficiency  of  the 
MAGTF  C4I  SoS;  users  will  be  more  effective  at  managing  and 
using  information.  IfMC3T fails,  users  will  be  forced  to 
operate  in  their  current  ad-hoc  fashion. 

6.  Your  Organization: 

a.  How  long  has  your  organization  been  in  operation?  MCTSSA  has  been  in 
operation  since  the  1970s. 

b.  Does  your  organization  follow  INCOSE,  IEEE,  ANSI  or  other  process 
standards?  MCTSSA  does  not  have  a  defined,  repeatable  set  of  process 
standards.  MC3T  is  an  attempt  to  introduce  process  standards. 

c.  How  long  have  you  operated  (sold,  created,  supported)  your  SoS?  Since 


d.  Is  there  anything  you  would  like  to  add  about  your  organization  or  SoS? 
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JTEM  CTM  Response  to  Questionnaire  Version  1 

1.  SoS  Definition: 

a.  What  is  your  SoS?  A  Joint  SoS  test  process. 

b.  Where  is  it  used?  JTEM  will  work  with  Live,  Virtual,  and  Constructive 
elements  on  joint  distributed  environment  ranges. 

c.  What  does  it  do?  JTEM  will  define  a  Joint  SoS  test  process.  . 

d.  What  are  the  high-level  components  of  your  SoS?  DOTMLPF 

e.  Are  components  created  or  managed  independently,  or  are  they  managed 
centrally?  JTEM  will  be  created  centrally  (DOT&E),  with  inputs  from  the 
JTEM  COL  As  the  JTEM  process  becomes  included  in  JCIDS  and  other 
acquisition  processes,  it  will  be  managed  and  executed  independently. 

f.  Was  your  SoS  built  to  a  systems  architecture  model,  as  an  automobile 
might  be,  or  did  it  evolve,  as  the  Internet  has?  JTEM  will  be  built  to  a 
systems  architecture. 

2.  Requirements: 

a.  Who  creates  requirements  for  your  SoS?  JTEM  is  creating  SoS  testing 
requirements,  based  on  COL  input.  Note  the  JTEM  charter,  the 
transformation  roadmap.  At  Rock  Drills  a  gap  was  discovered  in 
requirements  for  Joint  mission  test  requirements. 

b.  Are  requirements  generated  within  or  outside  your  organization?  JTEM 
requirements  are  generated  internally,  other  than  those  requirements  that 
come  from  DOD  acquisition  instructions  (JCIDS,  others). 

c.  Does  the  requirements  generator  have  any  authority  over  construction, 
operation,  or  maintenance  of  the  SoS?  JTEM  does  not  have  any  authority 
over  construction,  operation,  or  maintenance  of  the  SoS,  first,  because 
JTEM  is  a  limited-duration  project,  and  second,  because  JTEM  will  define 
a  process  for  use  by  others. 

d.  Do  requirements  for  the  SoS  change  over  time?  As  the  JTEM  COL 
matures,  and  JTEM  events  are  assesses,  SoS  (JTEM)  requirements 
change. 

e.  Are  requirements  documented?  ITEM  is  defining  requirements  as  they 
are  identified. 

f.  Can  I  review  your  requirements?  Yes,  see  latest  documentation  (Draft) 
Rock  Drill  Event  Final  Report. 

3.  Validation  and  Verification  (V&V): 

a.  Is  validation  (“does  it  do  the  right  things?”)  or  verification  (“does  it 
perform  to  specifications?”)  required  for  your  SoS,  or  for  component 
systems?  JTEM  has  conducted  V&V  through  rock  drills  (tabletop 
exercises)  to  determine  if  recommended  processes,  at  their  current  state, 
work.  Rock  drills  also  expose  gaps  and  seams  between  current  processes. 
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b.  Who  performs  V&V  for  your  SoS?  Is  this  an  internal  or  external 
organization?  V&V  (rock  drills)  was  conducted  internally,  with 
participants  from  almost  all  services  and  many  agencies. 

c.  What  are  sources  of  V&V  requirements:  are  they  derived  from  system 
requirements,  user  requirements,  or  other  sources?  System  requirements, 
doctrine,  regulations,  and  user  requirements. 

d.  What  are  the  consequences  of  V&V  failure?  “Failure”  of  V&V,  in  the 
early  stages  of  JTEM  maturation,  is  very  unlikely,  because  JTEM  is 
identifying  requirements,  testing  iteratively,  and  exposing  gaps  and  seams 
in  models  and  processes.  If  a  current  version  of  JTEM  was  to  fail,  it 
would  provide  more  information  for  analysis,  and  be  the  precursor  to 
updated  models  and  processes. 

e.  How  is  V&V  performed  for  your  SoS?  See  (Draft)  Rock  Drill  Event  Final 
Report  for  detail. 

f.  If  components  of  the  SoS  change,  is  testing  performed?  Yes. . 

g.  Can  I  view  your  test  procedures  and  results  reports?  See  (Draft)  Rock 
Drill  Event  Final  Report  for  detail.  . 

h.  Is  there  a  process  to  define: 

a.  test  environment, 

b.  test  criteria, 

c.  test  procedures, 

d.  test  conduct,  and 

e.  results  reporting? 

There  will  be  a  series  of  test  events  (USAF?)  this  year,  and  a 
follow-on  next  year  with  FCS  [Future  Combat  System]. 

4.  Support: 

a.  Is  your  SoS  supported  by  an  internal  or  external  organization?  JTEM  is 
supported  internally.  On  execution/implementation,  JTEM  will  be 
supported  by  using  organizations,  which  may  utilize  internal  or  external 
resources. 

b.  How  is  your  SoS  maintained?  On  execution/implementation,  JTEM  will 
be  supported  by  using  organizations,  which  may  utilize  internal  or 
external  resources. 

c.  Is  maintenance  performed  by  an  internal  or  external  organization?  On 
execution/implementation,  JTEM  will  be  supported  by  using 
organizations,  which  may  utilize  internal  or  external  resources. 

d.  Is  the  SoS  subject  to  change?  JTEM  is  evolving  and  expects  to  continue  to 
evolve. 

e.  How  is  the  configuration  of  the  SoS  initially  recorded,  how  is  it  checked, 
and  how  is  it  controlled?  JTEM  will  change  as  the  System  Under  Test 
(SUT)  changes.  User  organizations  will  control  configuration  through 
current  and  future  internal  processes. 
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/  If  part  of  the  SoS  is  inoperative,  how  do  users  request  service?  On 
execution/implementation,  JTEM  will  be  supported  by  using 
organizations,  which  may  utilize  internal  or  external  resources. 

g.  How  are  users  trained  on  functions  of  the  SoS?  On 
execution/implementation,  JTEM  will  be  supported  by  using 
organizations,  which  may  [generate?]  current  or  future  processes  for  user 
training. 

h.  Are  there  user,  support,  or  other  training  resources  I  can  view?  There  may 
be  access  to  limited  training  information  on  the  conduct  of  rock  drills. 

i.  How  are  support  personnel  trained?  On  execution/implementation,  JTEM 
will  be  supported  by  using  organizations,  which  may  [generate?]  current 
or  future  processes  for  user  training. 

5.  SoS  Operations: 

a.  What  is  the  life  cycle  of  your  SoS?  JTEM,  by  charter,  will  only  last  until 
[2009?].  After  this  point,  JTEM  will  be  supported  by  using  organizations. 

b.  Does  your  SoS  operate  continuously  or  intermittently?  TheJTEM process 
will  operate  continuously  as  SoS  are  under  test.  . 

c.  How  long  do  you  expect  your  SoS  to  function?  JTEM  is  ivolving  and 
expects  to  continue  to  evolve. 

d.  Does  your  SoS  have  documentation?  See  (Draft)  Rock  Drill  Event  Final 
Report  for  detail.  See  also  other  JTEM  documents. 

e.  Can  I  view  the  documentation?  See  (Draft)  Rock  Drill  Event  Final  Report 
for  detail.  See  also  other  JTEM  documents. 

f.  Is  there  an  alternative  or  backup  to  your  SoS?  Current  disparate, 
fragmented,  and  ad  hoc  testing. 

g.  What  are  the  consequences  (risk  of  injury  or  loss  of  life,  risk  of  damage  or 
loss  of  equipment,  risk  of  financial  loss,  other  risk)  of  SoS  failure?  Higher 
resource  use  (cost)  for  testing  due  to  “reinvention  ”  of  test  processes  for 
events;  lower  confidence  in  capability  of  SoS  to  achieve  capability; 
increased  possibility  of  Invalid  assessment  of  SoS  suitability  for  use 
(milestone  decisions). 

6.  Your  Organization: 

a.  How  long  has  your  organization  been  in  operation?  JTEM  was  chartered 
in  2006. 

b.  Does  your  organization  follow  INCOSE,  IEEE,  ANSI  or  other  process 

standards? _ . 

c.  How  long  have  you  operated  (sold,  created,  supported)  your  SoS?  JTEM 
was  chartered  in  2006 

d.  Is  there  anything  you  would  like  to  add  about  your  organization  or  SoS? 
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QUESTIONNAIRE  VERSION  2. 


JITC  Response  to  Questionnaire  Version  2 


1.  SoS  Assessment  Planning 

a.  Does  planning  for  an  assessment  have  a  relationship  to  the  scope  of  the 
assessment?  Yes.  For  larger,  more  complex  systems,  or  systems  requiring 
multiple  test  venues  we  develop  an  ICEP  (Interoperability  Certification 
Evaluation  Plan  —  basically,  a  plan  for  managing  the  testing). 
Interoperability  Test  Plans  (ITPs)  provide  detailed  plans/procedures  for 
individual  tests.  Larger  programs  will  also  have  a  TEMP  or  similar 
document.  For  many,  if  not  most  assessments,  JITC  leverages  off  testing 
conducted  by  others,  including  use  of  other ’s  test  plans/procedures. 
Usually,  this  involves  reviewing  OT  [Operational  Test]  plans  to  ensure 
adequate  data  is  collected  for  JITC  to  perform  an  interoperability 
evaluation. 

b.  Does  your  organization  have  a  planning  template  for  assessments?  Not 
sure  what  is  meant.  We  have  guidance/policy,  there  is  also 
guidance/policy  in  CJCSI  6212.01,  etc. 

c.  Does  your  organization  use  plans  or  results  of  previous  assessments  to  aid 
in  planning  for  new  events?  Absolutely! 

2.  SoS  Assessment  Resources: 

a.  How  are  resources  required  to  conduct  the  assessment  (time,  people, 
hardware,  software,  processes)  identified?  During  planning. 

b.  How  are  resource  gaps  identified  and  mediated?  Same. 

c.  How  are  resource  conflicts  identified  and  mediated?  Same.  Conflicts  with 
external  resources  (e.g.,  availability  of  interfacing  systems)  can  be  raised 
to  the  MCEB [Military  Communications  &  Electronics  Board 
/Interoperability  Test  Panel  or  Joint  Staff  (J-6),  if  necessary,  but  this  is 
rarely  required. 

d.  How  are  priorities  of  support  established:  This  was  covered  in  previous 
versions  of  6212,  however,  it  never  really  became  an  issue  by  itself.  See 
excerpt  from  CJCSI  62 12.0 1C  below: 

“DISA[Defense  Informatin  Systems  Agency]  (JITC)  uses  the  following 
organizational  prioritization  for  testing,  assessing,  and  certifying 
interoperability:  (i)  joint  IT  and  NSS  [National  Security  Systems]  systems 
that  support  the  unified  commands,  (ii)  joint  IT  and  NSS  systems  that  are 
acquired  by  the  Services,  and  (Hi)  systems  that  are  acquired  by  the 
Defense  agencies. 

The  order  for  functional  prioritization  is:  (i)  strategic  warning  and 
communication  systems  that  support  the  unified  commands,  the  Secretary 
of  Defense,  and  the  Commander-in-Chief;  (ii)  tactical  systems  that  support 
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the  unified  commands;  (iii)  C2  systems  that  support  the  unified 
commands;  and  (iv)  Combat  service  support  systems  that  support  the 
unified  commands.  ” 


(For  some  testing  at  JITC,  such  as  standards  conformance  and  DSN 

[Defense  Switched  Network],  there  is  a  simple  FIFO  {First  In,  First  Out] 

queue  approach. 

i.  External  agency  priorities? 

ii.  Higher  Head  Quaters  priorities? 

iii.  Other? 

3.  SoS  Components  Evaluated 

a.  How  are  components  of  the  SoS  identified  for  participation  in  the 
assessment?  DoD  4630  and  5000  series,  CJCSI  6212.01,  etc.  require 
systems  to  be  interoperable,  have  requirements  certified,  and  receive  JITC 
Joint  Interoperability  Test  Certification.  Every  system/system  component 
(even  legacy,  unless  granted  a  waiver)  is  in  effect  “nominated”  by  birth  in 
the  DoD  community.  J-6I  certified  capabilities  documents  identify 
systems/system  components  (e.g.,  in  S V  [Systems  View]- 1/2  architecture 
products). 

b.  What  organization(s)  nominate  components  for  assessment?  The 
PM/sponsor  is  responsible  for  ensuring  that  interfacing  systems 
participate  in  testing  ( see  6212). 

c.  How  are  conflicts  resolved?  MCEB/ITP,  J-6I. 

4.  SoS  Performance  Requirements 

a.  How  are  SoS  performance  requirements  identified?  Integrated 
architecture  products  and  associated  information  (e.g.,  an  interface  may 
have  an  interface  control  document).  Ideally,  architecture  information 
from  interfacing  systems  would  be  used  to  validate  requirements  for  a 
system.  There  are  also  overarching  requirements  such  as  the  NCOW[Net- 
Centric  Operations  Warfare-  Reference  Model]  RM  (including  DoD  Data 
Strategy),  GIGICD,  GIG  Architecture,  etc. 

b.  What  organization(s)  identify  SoS  performance  requirements? 

iv.  Subject  Matter  Experts? 

v.  Test  Agency? 

vi.  Program  Office? 

vii.  End  user  community? 

viii.  Doctrine  organizations? 

ix.  Joint  or  Service  organizations? 

x.  Other?  All  of  the  above,  to  some  extent.  PMs/sponsors  (e.g.,  in  the 
Army  the  combat  developers  generate  initial  requirements)  create 
requirements  that  are  reviewed  by  the  JS,  users,  COCOMs,  DISA, 
JITC,  and  other  organizations. 

c.  When  are  performance  requirements  identified: 
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xi.  Before  assessment  request  is  accepted? 

xii.  After  approval  of  assessment? 

xiii.  Before  assessment  planning  stage  starts? 

xiv.  Other?  6212  requires  that  requirements  be  testable  and 
measurable  from  the  start,  and  that  test  planning  start  early  in  the 
life-cycle.  However,  sometimes  requirements  are  not 
finalized/certified  as  early  as  desirable,  and  sometimes 
requirements  are  not  as  testable  &  measurable  as  they  should  be. 

d.  How  are  conflicting  requirements  recorded  and  resolved?  Conflicts 

should  be  resolved  during  the  document  development  and  review  process. 
If  conflicts  are  identified  later,  J-61  may  be  engaged  to  resolve  conflicts. 

5.  SoS  Performance  Criteria 

a.  When  are  SoS  performance  criteria  (i.e.  speed,  accuracy,  timeliness, 
authenticity,  repeatability. . .)  identified?  When  the  requirements  are 
developed,  although,  as  noted  above,  sometimes  this  information  must  be 
derived  from  supporting  documents.  (Integrated  architecture  products 
include  timeliness,  etc.) 

b.  If  performance  criteria  are  not  identified  before  an  evaluation  of  the  SoS  is 
requested,  what  happens?  JITC  can  evaluate  a  system  based  on  draft 
requirements,  and  produce  an  interoperability  assessment.  After  the 
requirements  are  certified  the  assessment  letter  could  be  upgraded  to  a 
certification,  provided  the  system  passes  and  there  are  no  changes  to  the 
requirements  that  would  require  additional  data  not  collected  during 
previous  testing.  Requirements  that  are  not  critical,  may  be  addressed  by 
a  status  of  “not  tested,  ”  if  other  data  still  supports  an  assessment  of 
critical  items.  (Threshold  KPP  requires  only  joint  critical  requirements  to 
be  met.) 

c.  How  are  conflicting  requirements  resolved?  See  above. 

6.  SoS  Threshold  and  Objective  values. 

a.  How  are  threshold  and  objective  values  (i.e.  500  KPH  threshold,  700  KPH 
objective;  10  meter  Circular  Error  Probable  (CEP)  threshold,  5  meter  CEP 
objective;  less  than  10  seconds  threshold...)  identified?  NR-KPP has 
threshold/objective  values,  as  do  KIPs  and  other  requirements. 

b.  Who  participates  in  this  identification?  Same  as  for  requirements. 

c.  How  are  conflicting  requirements  resolved?  Ditto. 

7.  SoS  Test  Script 

a.  What  organization  creates  the  test  script?  Usually,  the  primary  test 
organization  (e.g.,  OTA[Operational  Test  Agency]);  JITC  reviews  test 
plans,  procedures,  scripts,  etc.  to  ensure  data  collected  is  adequate  to 
address  interoperability  evaluation. 
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b.  What  stakeholders  approve  the  test  script?  Generally,  the  testing 
organizations  and  others  involved  in  the  various  working  groups  (varies 
by  type,  size  of  system). 

c.  How  are  conflicts  resolved?  By  test  working  groups.  Interoperability 
issues  can  always  be  raised  to  the  MCEB/ITP  or  J-6,  although  that  would 
be  less  likely  for  issues  involving  very  technical  details  of  a  script. 

8.  SoS  Configuration: 

a.  When  is  the  SoS  assessment  configuration  defined?  JITC  tests  to  the 
approved  Information  Assurance  (I A)  configuration.  The  configuration 
must  represent  a  realistic  operational  environment. 

b.  When  is  the  SoS  assessment  configuration  installed?  During  pre-test 
activities. 

c.  Who  can  authorize  changes  to  the  SoS  assessment  configuration,  and 
under  what  conditions?  The  approved  IA  configuration  must  be  used.  Any 
deviations  would  have  to  be  reported  as  testing  limitations,  including  an 
assessment  of  the  impact  they  may  have  on  interpretation  of  results.  If 
changes  are  necessary  to  proceed  with  testing  (e.g.,  there  are 
showstoppers  that  would  make  continued  testing  a  waste),  the  impact  on 
previously  collected  data  must  be  assessed  and  regression  testing 
performed  as  needed  to  revalidate  earlier  results. 

9.  What  are  milestones  in  the  planning,  conduct,  and  reporting  of  an 
assessment: 

a.  Planning? 

b.  Conduct? 

c.  Reporting? 

d.  Other?  JITC  uses  a  basic  4-step  process.  Requirements  (identify 
requirements,  ensure  they  are  J-6  certified,  testable/measurable,  etc.); 
plans  (ICEP,  ITPs,  [Interoperability  Certification  Evaluation  Plan, 
Interoperability  Test  Plan]  etc.);  conduct  (more  often  monitoring  other’s 
testing  and  collecting  data);  reporting  (detailed  test  reports,  assessments, 
certifications,  etc.).  JITC,  per  DoDI  4630.8,  also  provides  input  to  the 
OTRR  [Operational  Test  Readiness  Review].  Standards  conformance  or 
other  assessments  may  also  enter  into  the  overall  testing  program  at 
various  stages. 

10.  Conduct  of  Performance  Assessment 

a.  Who  executes  the  test  script: 

i.  Contractors? 

ii.  government  civilians? 

Hi.  other?  A  realistic  environment  usually  requires  that  the  typical 
user  performs  the  operations.  Typical  users  may  be  military, 
civilians,  contractors,  etc.  Some  types  of  systems/situations  may 
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require  supplementing  the  typical  user  with  other  types  of 
personnel,  as  long  as  this  does  not  invalidate  test  results. 

b.  Who  supervises  the  conduct  of  the  assessment: 

iv.  Test  Agency  personnel? 

v.  Program  Office  personnel? 

vi.  Combination/Other?  Lead  test  organization,  usually  OTA.  JITC 
“supervises  ”  (or  monitors)  testing  as  need  to  ensure 
interoperability  data  is  adequate. 

11.  Assessment  Results 

a.  Who  records  assessment  results: 

i.  Test  Agency  personnel? 

ii.  Program  Office  personnel? 

iii.  Combination/Other?  Same  as  supervising,  although  “operators  ” 
record  some  data  -  data  collection  is  automated  to  the  extent 
practicable. 

b.  How  are  results  expressed? 

i.  Accurate  until  SoS  components  change?  Joint  interoperability 
Test  Certificates  state  “This  certification  expires  upon  changes 
that  affect  interoperability,  but  no  later  than  three  years  from  the 
date  of  this  memorandum.  ”  Changes  that  affect  interoperability 
could  include  any  or  all  of  the  following:  hardware  and/or 
software  changes  to  the  system,  DOTMLPF  changes  to  the  system, 
or  similar  changes  to  interfacing  systems. 

ii.  Accurate  for  a  period  of  time?  Three  (3)  years,  because  most 
systems  have  minor  updates  (sometimes  frequently)  and,  even  if  the 
system  under  test  does  not  change,  interfacing  systems  are  likely  to 
have  changed,  or  requirements  may  have  (should  have)  changed, 
etc. 

iii.  Other? 

c.  How  are  results  expressed: 

i.  Pass/Fail?  Yes 

ii.  Numerically  scored?  Yes 

iii.  Narrative  description  (without  score)?  Yes 

iv.  Other?  JITC  reports  an  overall  status  (e.g.,  threshold  NR-KPP 
met);  an  overall  status  of  the  elements  of  the  NR-KPP  (net-centric, 
information  exchange,  KIP,  IA,  and  DISR  [DoD  IT  Standards 
Registry] compliance) ;  status  of  interfaces  (certified,  not  certified, 
not  tested),  and  more  detailed  status  on  net-centric  requirements 
(enterprise  (NCES  [Net-Centric  enterprise  Services]  /COIflevel 
services/data),  information/data  exchanges  (OV [Operational 
View]-3/SV-6),  KIP  and  DISR  [DoD  IT  Standards  Registry] 
compliance,  and  IA  related  compliance.  Depending  on  the 
size/complexity  of  requirements,  some  more  detail  may  be  provided 
in  the  certification  letter  (consisting  of  a  memo  and  summary 
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sections),  or  in  a  detailed  test  report  if  too  voluminous  to  include 
in  the  certification  letter.  JITC  reports  the  degree  to  which 
requirements  are  met  (e.g.,  all  critical  requirement  met,  all 
requirements  met)  and  the  expected  operational  impact  (none, 
minor,  moderate,  major,  or  critical)  of  any  discrepancies. 

12.  Communications: 

a.  How  frequently  are  results  reported  Varies  with  size/complexity,  length  of 
testing,  visibility  of  program,  etc.  Quick  Look  reports/briefings  are 
sometimes  produced,  informal  testing  status  can  be  produced  daily,  etc. 
Detailed  test  reports  are  produced  after  testing  (sometimes  by 
organizations  other  than  JITC);  certification  letters  are  finalized  and 
distributed  after  results  have  been  analyzed. 

xv.  Under  predefined  conditions? 

xvi.  On  demand? 

xvii.  Daily/weekly? 

xviii.  At  the  conclusion  of  the  assessment? 

b.  Is  there  a  separate  assessment  report?  For  large/complex  systems,  yes. 

c.  Who  controls  distribution  of  the  assessment  results? 

Distribution  of  JITC  interoperability  certifications  is  specified  in  6212. 
Test  reports  produced  by  JITC  are  provided  to  the  sponsor  and  other 
interested/authorized  parties. 

13.  Organization  Performance  Evaluation  Metrics: 

a.  Do  you  measure  equipment  resources  used  per  event? 

If  equipment  is  provided/controlled  by  JITC,  use  is  monitored  to  some 
extent.  Resources  that  are  periodically  used  to  support  testing  (part  of  an 
existing  lab  or  general  test  capability)  are  not  necessarily  tracked  by  each 
individual  test  event  (and  some  resources  may  be  shared  by  more  than  one 
system  during  a  test  event).  An  example  is  NIPRNET  [Non-Secure 
Internet  Protocol  Router  Network]  access.  It  also  depends  on  the  nature 
of  the  testing,  overall  test  configuration  (many  involve  distributed  test 
resources  provided  by  a  number  of  organizations),  etc. 

b.  Personnel  labor  hours  used  per  event?  Yes. 

c.  Duration  of  processes  and  entire  event?  Yes. 

d.  Other  resources  used  in  planning,  conducting,  and  reporting  an  event? 
Much  of  the  interoperability  testing  is  piggybacked  on  testing  performed 
mostly  by  others,  usually  OT  organizations.  JITC  tracks  resources  used 
for  its  portion  of  planning,  conduct,  and  reporting. 

14.  SoS  Performance  Metrics: 

a.  Does  your  organization  evaluate  the  performance  of  the  SoS  after  testing? 
Whenever  possible,  JITC  continues  to  observe  performance  by 
participation  in  numerous  exercises,  other  interoperability  testing  that 
includes  previously  tested  systems  as  participants,  reports  from  the  field 
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( from  various  sources),  etc.  When  necessary,  JITC  reviews  the  previously 
determined  interoperability  status  to  determine  if  the  status  has  changed, 
possibly  requiring  additional  testing  and  updating  any  previous 
certification  letters.  Systems  with  known  discrepancies,  including  failure 
to  obtain  an  interoperability  certification,  may  be  reported  to  J-6I  and/or 
the  MCEB/ITP  for  additional  action  (there  is  a  J-6I  delinquency 
program/ITP  watch  list).  JITC/J-6I  also  track  expiring  certifications; 
MCEB  /ITP  tracks  ICTOs  [Interim  Certification  to  Operate]  (part  of 
qualifying  for  an  ICTO  is  a  plan  for  obtaining  JITC  certification,  usually 
within  a  year). 

b.  Does  your  organization  record  comments  or  queries  from  end  users?  After 
fielding,  JITC  has  a  24/7  Hotline  to  address  inquiries/issues  from  end 
users  (and  anyone  else,  for  that  matter).  Appropriate  action  is  taken, 
including  deploying  personnel  to  help  resolve  interoperability  problems. 
(During  testing,  see  below.) 

c.  Does  your  organization  provide  feedback  to  program  developers  on  end 
user  comments?  JITC  does  not  normally  collect  end  user  comments  after 
a  system  is  fielded,  however,  appropriate  organizations  are  notified  of 
serious  interoperability  issues  as  they  are  identified.  (See  above.)  User 
comments  during  testing  are  normally  recorded  and  provided  to  the 
sponsor  as  part  of  the  SOP  for  most  testing.  When  there  are  discrepancies 
during  testing,  JITC  provides  an  “expected  operational  impact”  based  on 
input  from  the  user  community. 
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APPENDIX  C.  QUALITY  OF  PLANNING  OUTPUTS 
QUESTIONNAIRE 


This  appendix  consists  of  three  sections.  First,  a  description  of  the  research  that 
went  into  the  methodology  behind  the  Quality  of  Planning  Outputs  Questionnaire  is 
provided.  Second,  the  actual  Quality  of  Planning  Outputs  Questionnaire  is  provided. 
Third,  tables  of  the  results  are  provided,  including  a  table  with  final  calculation  of  the  raw 
scores. 

EVALUATION  MEASURE:  QUALITY  OF  PLANNING  OUTPUTS 

The  team  considered  two  scales  to  process  SME  questionnaire  responses. 

Osgood’s  semantic  differential  scale  [Pershing,  2000:1-7]  measures  subjective 
feelings  indirectly  by  comparing  extremes  of  the  scale  using  words  of  opposite  meaning. 
Respondents  consider  a  list  of  paired  opposite  adjectives  on  a  continuum  of  5  to  1 1  points 
and  recorded  their  feelings,  likes,  perceptions  on  the  scale.  The  scale  produces  interval 
data  that  can  be  used  in  statistical  analysis.  Concise  directions,  which  are  understood  by 
all  respondents  in  the  same  way,  are  mandatory  for  effective  use.  The  varied  background 
of  the  respondents  was  valuable  for  their  experience,  but  made  this  scale  less  effective  for 
this  questionnaire. 

A  Likert  [Mitchell  and  Jolley,  2001:  485-487]  scale  is  useful  because  responses 
can  be  summed  to  provide  an  overall  score  from  each  respondent.  Statistical  analysis, 
typically  a  t-test,  can  be  used  to  determine  the  validity  and  accuracy  of  the  responses. 
Likert  scales  are  often  constructed  of  two  positive  responses,  two  negative  responses,  and 
a  neutral  response.  The  team  constructed  a  scale  of  two  positive  responses  and  two 
negative  responses,  and  values  from  1  (worst)  to  4  (best),  in  order  to  avoid  any  undecided 
or  “fence  sitter”  answers. 

Determining  the  Score  for  the  Evaluation  Measure  Quality  of  Planning 
Outputs 

The  team  created  a  questionnaire  consisting  of  four  questions  that  covered  key 
areas  of  what  constitutes  the  quality  of  planning  outputs.  Each  question  also  contained  a 
criterion  in  order  to  help  respondents  differentiate  between  a  value  of  1  to  a  value  of  4. 

The  team  administered  the  questionnaire  to  one  SME  from  MCTSSA  and  two 
SMEs  from  NAWC  China  Lake.  Responses  are  recorded  in  Table  21,  Table  22,  and 
Table  23.  The  Table  22  response  is  skewed  with  respect  to  the  responses  in  Table  21  and 
Table  23.  The  Table  22  respondent  indicated  the  FCB  and  SCR  alternatives  were  not  as 
clearly  understood,  and  their  ability  to  produce  quality  planning  products  appeared  to  be 
reduced. 
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The  raw  score  was  calculated  by  summing  the  scores  for  each  alternative  from  the 
respondents.  The  overall  score  for  the  evaluation  measure  was  calculated  by  dividing  the 
raw  score  by  the  number  of  questions,  which  were  four.  The  equation  for  overall  score  is 

overall  score  =  raw  score  /  #  questions.  (5) 

In  order  to  use  this  score  with  the  weights  utilized  in  AoA,  the  raw  index  was 
calculated  by  dividing  the  overall  score  by  the  number  of  respondents.  Thus,  a  raw  index 
will  contain  a  value  from  1  to  4.  The  equation  for  raw  index  is 

raw  index  =  overall  score  /  #  SMEs.  (6) 

The  raw  index  was  used  in  AoA  to  compare  quality  of  planning  outputs  for  each 
alternative. 

Participants 

•  The  SME  respondent  from  MCTSSA  was  Mr.  Scot  Hoesly;  the  SME  respondents 

from  NAWC  China  Lake  were  Mr.  Stephan  Bussell  and  Mr.  Robert  Mount. 
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EVALUATION  MEASURE:  QUALITY  OF  PLANNING  OUTPUTS 
QUESTIONNAIRE 


Question  #1:  Ability  to  Create  Planning  Documents  -  Does  the  alternative 
develop  planning  products  (test  plans,  test  procedures)  for  C4I  SoS  evaluations? 

Objective:  Planning  products  are  produced  each  time  the  process  is  executed. 
Criteria: 


Criterion 

Score 

The  alternative  always  produces  planning 
products  for  evaluating  C4I  SoS. 

4 

The  alternative  often  produces  planning 
products  for  evaluating  C4I  SoS. 

3 

The  alternative  sometimes  produces 
planning  products  for  evaluating  C4I  SoS. 

2 

The  alternative  never  produces  planning 
products  for  evaluating  C4I  SoS. 

1 

Question  #2:  Quality  of  Planning  Documents  -  Does  the  alternative  deliver 
planning  documents  that  conform  to  standards  established  by  relevant  official 
directives  and  guidance?  (See  examples  of  standards  below) 

Objective:  The  planning  products  conform  to  standards. 

Criteria: 


Criterion 

Score 

The  alternative  produces  planning 
documents  that  conform  to  standards 
established  by  all  Services  affected  by  the 
SoS 

4 

The  alternative  produces  planning 
documents  that  conform  to  the  majority  of 
standards  in  directives 

3 

The  alternative  produces  planning 
documents  that  conform  to  standards  in  a 
single  Service’s  directives 

2 

Planning  documents  do  not  conform  to 
established  standards 

1 
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Question  #3:  Usability  of  Planning  Documents  -  Does  the  alternative  produce 
planning  products  that  can  be  used  for  C4I  SoS  evaluations? 

Objective:  The  planning  products  can  be  used  easily  in  an  organization  tasked 
with  designing  and  executing  SoS  evaluation. 

Criteria: 


Criterion 

Score 

The  alternative  provides  planning  products 
that  are  completely  usable  for  executing 

C4I  SoS  evaluations 

4 

The  alternative  provides  planning  products 
that  are  usable  for  executing  C4I  SoS 
evaluations 

3 

The  alternative  provides  planning  products 
that  are  somewhat  usable  for  executing  C4I 
SoS  evaluations 

2 

The  alternative  provides  planning  products 
that  are  not  usable  for  executing  C4I  SoS 
evaluations 

1 

Question  #4:  Planning  Documents  determination  of  Capability  -  Does  the 
alternative  produce  outputs  that  measure  C4I  SoS  capabilities? 

Objective:  The  planning  products  are  effective  for  evaluating  C4I  SoS 
capabilities. 

Criteria: 


Criterion 

Score 

Planning  documents  ensure  execution  of 
the  evaluation  will  measure  all  desired 
capabilities  of  the  SoS 

4 

Planning  documents  ensure  evaluation  of 
most  of  the  desired  capabilities  of  the  SoS 

3 

Planning  documents  ensure  evaluation  of 
some  of  the  desired  capabilities  of  the  SoS 

2 

Planning  documents  ensure  evaluation  of 
basic  data  exchange  and  technical 
interfaces  of  the  SoS 

1 
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QUESTIONNAIRE  RESULTS. 

Table  21,  Table  22,  and  Table  23  display  the  results  from  the  questionnaires  for 
each  of  the  SMEs.  Table  24  provides  the  raw  index  which  was  used  during  the  AoA  in 
determining  the  quality  of  planning  outputs  for  each  alternative. 


Alternative 

Question 

1 

Question 

2 

Question 

3 

Question 

4 

Score 

FEDOS 

3 

3 

4 

2 

12 

MC3T 

3 

2 

3 

3 

11 

JTEM  CTM 

3 

3 

4 

3 

13 

SCR 

4 

4 

3 

3 

14 

FCB 

4 

3 

4 

4 

i  15 

Table  21.  Scot  Hoesly’s  Responses  to  Quality  of  Planning  Outputs 
Questionnaire. 


Alternative 

Question 

1 

Question 

2 

Question 

3 

Question 

4 

Score 

FEDOS 

4 

4 

4 

4 

16 

MC3T 

4 

4 

4 

4 

16 

JTEM  CTM 

4 

4 

4 

4 

16 

SCR 

2 

2 

2 

2 

8 

FCB 

1 

1 

1 

1 

4 

Table  22.  Stephan  Bussell’s  Responses  to  Quality  of  Planning  Outputs 
Questionnaire. 


Alternative 

Question 

1 

Question 

2 

Question 

3 

Question 

4 

Score 

4 

3 

2 

1 

4 

3 

3 

2 

12 

4 

3 

3 

2 

12 

SCR 

4 

3 

3 

4 

14 

FCB 

4 

3 

3 

4 

14 

Table  23.  Robert  Mount’s  Responses  to  Quality  of  Planning  Outputs 
Questionnaire 
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Alternatives 

Raw 

Score 

Raw 

Index 

FEDOS 

38 

3.167 

MC3T 

39 

3.250 

JTEM  CTM 

41 

3.417 

SCR 

36 

3.000 

FCB 

33 

2.750 

Table  24.  Table  of  Raw  Index  Scores. 
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APPENDIX  D.  DESIGN  ALTERNATIVES  DETAILS 


This  Appendix  discusses  alternatives  generation  and  feasibility  screening  of  the 

Design  Alternatives  phase  in  further  detail.  This  Appendix  begins  with  detailed 

descriptions  of  the  alternatives  generated  during  the  alternatives  generation  phase,  and  is 

concluded  with  a  detailed  description  of  the  feasibility  screening  process  that  was  used  to 

select  feasible  alternatives  for  the  JC3M  project. 

GENERA  TED  AL  TERN  A  TIVES 
Exhaustive  Alternative 

The  “Exhaustive”  approach  embodied  the  theme  of  doing  everything,  without 
limiting  the  evaluation  to  specific  capabilities,  or  user-directed  systems  or  functions. 

This  approach  would  evaluate  every  component  of  the  SoS,  every  function,  every  system 
under  test,  every  capability,  and  every  component  of  the  individual  systems.  The  theory 
was  that  by  exhaustive  testing,  data  would  be  generated  for  every  evaluation  measure, 
and  a  complete  evaluation  of  the  SoS  would  be  provided. 

User  Defines  Alternative 

The  “User  Defines”  approach  was  based  on  the  user/stakeholder  determining  the 
functions,  capabilities,  and  components  of  the  SoS  under  evaluation.  This  approach 
would  evaluate  only  those  components  of  the  SoS,  systems  under  test,  component 
systems,  and  capabilities  requested  by  the  user.  The  theory  was  that  by  selective  testing, 
the  time  required  to  plan  an  evaluation  would  decrease,  the  user  would  get  a  selected 
review  of  capabilities  and  functionality,  and  performance  measures  would  accurately 
reflect  the  capabilities  under  review. 

“Do  No  Harm”  Alternative 

The  “Do  No  Harm”  approach  was  based  on  the  use  of  J1TC  to  determine  the 
capabilities,  functions  and  components  of  the  SoS  under  evaluation.  This  approach  would 
evaluate  only  those  components  of  the  SoS,  systems  under  test,  component  systems,  and 
capabilities  identified  by  J1TC,  in  their  current  processes,  as  required  to  demonstrate 
interoperability.  The  theory  was  that  by  selective  testing,  investigation  of  and 
consultation  on  performance  measures  would  be  reduced,  saving  time  in  planning  an 
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evaluation.  The  user  would  get  a  selected  review  of  capabilities  and  functionality  which 
would  be  directly  related  to  interoperability  evaluations. 

System  Anomaly  Report  (SAR)  Alternative 

The  “System  Anomaly  Report  (SAR)”  approach  was  based  on  determining  the 
functions,  capabilities,  and  components  of  the  SoS  under  evaluation  by  reference  to 
anomalies  or  concerns  received  from  the  user  community.  This  approach  would  evaluate 
only  those  components  of  the  SoS,  systems  under  test,  component  systems,  and 
capabilities  identified  by  the  user  community,  and  recorded  in  SARs,  as  problems  in 
previous  increments  of  the  SoS.  The  theory  was  that  the  user  community  would  define 
unacceptable  performance  of  the  SoS,  and  threshold  values  could  thus  be  deduced.  This 
would  enable  selective  testing,  so  that  investigation  of  and  consultation  on  performance 
measures  would  be  reduced,  saving  time  in  planning  an  evaluation.  End  users  would  get 
a  selected  review  of  capabilities  and  functionality  focused  on  reported  issues,  and 
performance  measures  would  accurately  evaluate  the  capabilities,  functions,  and 
components  of  interest. 

Deliberate  Method  Alternative 

The  “Deliberate  Method”  approach  was  based  on  the  most  effective  process  to 
determine  the  functions,  capabilities,  and  components  of  the  SoS  under  evaluation.  This 
approach  would  exhaustively  identify  those  components  of  the  SoS,  systems  under  test, 
component  systems,  and  capabilities  which  are  affected  or  are  under  review.  The  theory 
was  that  by  using  the  best  systems  engineering  approach  to  define  the  most 
discriminating  testing,  the  evaluation  would  be  more  effective  in  measuring  the 
performance  of  the  SoS,  systems  under  test,  or  capabilities,  and  the  stakeholders  would 
derive  greater  benefit  from  focused  testing.  This  approach  was  also  created  in  an  attempt 
to  define  the  highest  performance  alternative,  in  order  to  offer  a  markedly  different 
alternative  solution  to  compare  to  other  alternatives. 
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Capabilities  Documentation  Alternative 

The  “Capabilities  Documentation”  approach  was  based  on  the  use  of  requirements 
documents  to  determine  the  functions,  capabilities,  and  components  of  the  SoS  under 
evaluation.  This  approach  would  evaluate  components  of  the  SoS,  systems  under  test, 
component  systems,  and  capabilities  that  have  performance  measures  defined  by  Joint 
Capabilities  Integration  and  Development  System  (JCIDS)  documentation,  such  as  an 
Initial  Capabilities  Document,  Capability  Development  Document,  or  Capability 
Production  Document.  The  theory  was  that  by  reviewing  JCIDS  documentation, 
investigation  of  and  consultation  on  performance  measures  would  be  reduced,  saving 
time  in  planning  an  evaluation;  the  SoS  would  be  evaluated  in  accordance  with  its 
documented  intended  purposes;  and  performance  measures  would  accurately  reflect  the 
capabilities  of  the  SoS. 

Program  Manager  Direction  Alternative 

The  “Program  Manager  Direction”  approach  was  based  on  the  acquisition 
Program  Manager,  responsible  for  the  SoS  or  system,  determining  the  functions, 
capabilities,  and  components  of  the  SoS  under  evaluation.  This  approach  would  evaluate 
only  those  components  of  the  SoS,  systems  under  test,  component  systems,  and 
capabilities  requested  by  the  PM.  The  theory  was  that  the  Program  Manager  would 
define  threshold  values,  and  that  by  selective  testing,  investigation  of  and  consultation  on 
performance  measures  would  be  eliminated,  saving  time  in  planning  an  evaluation.  The 
PM,  as  representative  of  the  acquisition  and  user  community,  would  get  a  selected 
review  of  capabilities  and  functionality,  and  performance  measures  would  accurately 
evaluate  the  issues  in  question. 

Test  Agency  Direction  Alternative 

The  “Test  Agency  Direction”  approach  was  based  on  the  test  agency  determining 

the  functions,  capabilities,  and  components  of  the  SoS  under  evaluation.  This  approach 

would  evaluate  only  those  components  of  the  SoS,  systems  under  test,  component 

systems,  and  capabilities,  as  determined  by  the  test  agency,  to  bear  on  the  issues  under 

investigation.  The  theory  was  that  the  test  agency  was  an  objective  evaluator  of  both 

performance  measures  and  SoS  performance,  and  would  provide  an  unbiased,  accurate, 

and  methodical  evaluation  of  the  SoS.  Further,  time  could  be  saved  in  planning  an 
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evaluation  by  reducing  the  need  for  outside  consultation.  This  assumed  the  test  agency 
would  be  more  efficient  in  establishing  test  requirements  than  other  organizations.  This 
approach  was  also  created  in  an  attempt  to  offer  a  markedly  different  alternative  solution 
to  compare  to  other  alternatives. 

Change  Driven  Alternative 

The  “Change  Driven”  approach  was  based  on  the  test  agency  or  stakeholder 
determining  what  is  different  in  the  SoS  configuration  or  employment,  in  order  to 
determine  the  functions,  capabilities,  and  components  or  functions  of  the  SoS  to  be 
evaluated.  This  approach  would  evaluate  only  those  components  of  the  SoS,  systems 
under  test,  component  systems,  and  capabilities  which  have  changed,  in  order  to 
document  performance  variances.  The  theory  was  that  by  testing  only  changed  items, 
time  would  be  saved  in  planning  an  evaluation.  The  user  would  get  a  review  of 
capabilities  and  functionality  that  are  different  as  a  result  of  and  attributed  to  changes  to 
the  SoS.  Further,  performance  measures  would  accurately  reflect  the  capabilities  under 
review. 

FEASIBILITY  SCREENING 

After  reviewing  the  nine  alternatives  from  the  standpoint  of  feasibility  and 
similarity,  the  “User  Defines”  alternative  was  eliminated  from  further  consideration 
because  it  was  very  similar  to  the  “Program  Manager  Defines.” 

The  “Do  No  Harm”  alternative  was  eliminated  because  the  team  planned  to 
consider  the  performance  of  baseline  processes  from  other  organizations  (JTEM,  MC3T, 
MCTSSA)  from  the  start,  and  a  primary  goal  of  this  alternative  generation  process  was  to 
consider  two  entirely  new  alternatives,  rather  than  exclusively  baseline  processes. 

The  Capabilities  Documentation  alternative  was  eliminated  because  it  was  similar 
to  the  SAR  alternative  in  that  both  alternatives  rely  on  outside  documentation  for 
identification  of  threshold  values. 

The  revised  list  of  alternatives  was  left  as  Ad-Hoc,  SAR,  Deliberate  Method, 
Program  Manager  Direction,  and  Change  Driven. 
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One  process  of  screening  alternatives  with  multiple  criteria  [Blanchard  and 
Fabrycky,  1998:  567]  is  to  assign  weighting  factors  to  each  evaluation  measure,  score 
performance  on  each  evaluation  measure,  and  sum  scores  for  each  alternative,  which 
results  in  an  overall  scores.  This  method  was  not  used  because  all  five  alternatives  under 
consideration  were  theoretical,  and  there  was  no  measurable  data  to  compare.  Simulation 
or  modeling  of  the  alternative  was  not  employed,  because  the  goal  at  this  stage  was  not  to 
compare  the  performance  of  alternatives,  but  to  reduce  the  field  to  two  alternatives. 

Another  method  of  screening  alternatives  [Thuesen  and  Fabrycky,  2001 :  567]  is 
to  perform  paired  comparisons.  Paired  comparisons  contrast  the  performance  of  any  two 
alternatives  to  each  other  on  a  single  MOE,  to  determine  which  alternative  is  superior. 
Alternative  A  is  compared  to  B,  on  a  single  parameter,  with  the  result  a  ranking  of  the 
two  alternatives:  A  >  B.  The  process  is  repeated  (B>C,  C>D,  D>E. . .)  for  each 
alternative  and  MOE.  The  property  of  transitivity  (if  A  >  B  and  B  >  C,  then  A  >  C)  is 
applied  to  the  results  and  a  ranking  of  all  alternatives  can  be  generated.  Paired 
comparison  was  not  used  to  screen  the  alternatives  because  the  comparison  of  alternatives 
is  subjective,  depends  on  expert  evaluation  of  the  alternatives,  and  as  with  weighting 
factors,  there  was  no  measurable  data  to  compare.  Simulation  and  modeling  of  the 
alternative’s  performance  was  not  employed,  as  stated  above,  because  the  goal  was  to 
reduce  the  field  of  alternatives. 

Systematic  elimination  [Thuesen  and  Fabrycky,  2001:  569]  also  supports  the 
evaluation  of  the  alternatives  on  each  parameter,  by  eliminating  alternatives  that  do  not 
meet  a  minimum  level  of  acceptability.  Systematic  elimination  can  be  implemented  in  a 
restrictive  manner  by  stipulating  that  if  an  alternative  achieved  a  passing  score  on  each 
criterion,  the  alternative  would  be  retained  for  further  development.  An  unrestrictive 
implementation  of  systematic  elimination  would  keep  any  alternative  that  achieved  a 
passing  score  on  at  least  one  criterion.  A  moderate  approach  would  keep  alternatives  that 
meet  more,  or  fail  fewer,  criteria.  This  process  was  also  rejected  because  it  depends  on 
performance  scores,  which  do  not  exist  for  four  of  the  five  alternatives. 

In  cases  where  performance  data  are  not  known,  a  datum  design  comparison 

[Cross,  2000: 156]  can  be  conducted.  In  this  process,  each  alternative  is  compared, 
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parameter  by  parameter,  against  a  known  quality  solution.  The  result  of  each  comparison 
is  recorded  as  the  same  (=)  performance,  better  than  (+),  or  worse  than  (-)  the  baseline. 

At  the  completion  of  the  comparisons  the  positive  rankings  for  each  alternative  are 
summed,  as  are  the  negative  rankings,  and  an  overall  total  is  calculated  for  each 
alternative.  In  this  manner  the  qualitative  comparisons  of  performance  against  the 
baseline  were  converted  into  a  quantitative  ranking  of  alternatives.  The  overall  totals 
were  used  to  screen  out  the  untenable  alternatives  from  further  consideration. 

In  the  case  of  the  five  remaining  alternatives,  the  datum  design  comparison  was 
used,  and  the  parameters  for  scoring  were  drawn  from  the  evaluation  measures  that 
support  the  JC3M  functional  hierarchy.  The  advantage  of  using  the  datum  design 
comparison  is  that  the  team  understands  the  strengths  and  weaknesses  of  the  baseline 
alternative.  Thus,  by  comparing  the  proposed  new  alternatives  with  the  evaluation 
measures  against  the  baseline  the  team  could  speculate  whether  it  will  be  better,  worse,  or 
the  same  as  the  baseline  for  each  of  these  evaluation  measures.  Also,  these  evaluation 
measures  were  used  for  comparing  the  final  five  alternatives  in  the  AoA. 

For  each  evaluation  measure,  each  alternative  was  scored  against  the  MCTSSA 
(FEDOS)  baseline  datum.  The  scoring  criteria  were  as  follows:  (+)  the  alternative  was 
expected  to  perform  better  than  the  baseline,  (-)  the  alternative  was  expected  to  perform 
worse  than  the  baseline,  and  (=)  the  alternative  was  expected  to  perform  the  same  as  the 
baseline.  The  team  reviewed  the  known  performance  of  the  baseline  against  the 
projected  performance  of  the  alternative  process,  function  by  function.  This  ranking  was 
performed  initially  without  the  aid  of  an  SME.  After  evaluating  each  alternative,  the  (+) 
values  were  summed  and  the  (-)  values  were  subtracted,  to  calculate  a  final  score. 
Deliberate  Method  and  Change  Driven  alternatives  were  ranked  highest,  as  displayed  in 
Table  25. 
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Ad- 

Hoc 

SAR 

Deliberate 

Method 

PM 

Direction 

Change 

Driven 

Evaluation  Measures 

Percentage  of  Traceable  Thresholds 

= 

+ 

+ 

+ 

+ 

Percentage  of  Capabilities  Identified 

= 

+ 

+ 

+ 

= 

Days  to  Plan  Evaluation 

+ 

= 

= 

= 

+ 

Number  of  Traceable  Thresholds  Identified 

= 

+ 

+ 

+ 

+ 

Percentage  of  Shortfalls  Identified 

= 

= 

+ 

= 

= 

Quality  of  Planning  Outputs 

= 

= 

+ 

= 

= 

Number  of  C4I  Systems  Under  Test 

= 

= 

= 

= 

= 

Percentage  of  Interfaces  Identified 

= 

= 

+ 

= 

+ 

Total  + 

1 

3 

6 

3 

4 

Total  - 

0 

0 

0 

0 

0 

Overall  Total 

1 

3 

6 

3 

4 

Table  25.  JC3M  System  Alternatives  Datum  Design  Comparison  Matrix. 
Initial  Screening  of  Alternatives  against  Baseline  (FEDOS) 
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APPENDIX  E.  EVALUATION  MEASURES  DETAILS 


FUNCTIONS  AND  EVALUATION  MEASURES 

Evaluation  Measures  (EM)  were  developed  to  tie  to  each  function  of  the  JC3M 
system,  to  ensure  each  function  was  not  only  measurable,  but  potentially  capable  of 
serving  as  a  discriminator  in  considering  alternative  JC3M  solutions. 

In  the  following  review  of  the  functions  of  the  JC3M  system,  the  EMs  are 
described  in  terms  of  their  attributes,  using  a  methodology  described  by  the  U.S.  Army 
Training  and  Doctrine  Command  [Borman,  1993:  30], 

Definition  of  the  measure:  A  statement  of  the  measure  that  includes  directions  for 
computation.  Input  data  is  identified  for  use  in  evaluation  of  alternatives. 

Dimension  of  the  measure:  How  the  measure  is  expressed  in  terms  of  scale  of 
measurement  and  unit  of  measure.  The  scale  will  be  one  of  four  categories  [Pariseau, 
1994], 

Nominal  -  Nominal  data  is  “named”  data,  which  can  be  described,  but  cannot  be 
manipulated  arithmetically.  Preferences  cannot  be  implied  from  the  label.  Examples  of 
nominal  data  sets  include  red,  yellow,  blue;  eggs,  chickens,  cats;  Marines,  civilians, 
academics. 

Ordinal  -  Ordinal  data  is  “ordered”  data,  which  can  be  assigned  a  location  in  a 
sequence,  and  a  label  designating  that  location.  The  interval  between  the  values  is  not 
necessarily  consistent.  Ordinal  data  cannot  be  used  to  perform  mathematical 
combinations  or  operations,  other  than  a  comparison  of  rank.  Examples  of  ordinal  data 
sets  include  First,  Second,  Third;  A,  B,  C;  Superior,  Average,  Marginal. 

Interval  -  Interval  data  has  a  consistent  scale,  but  the  zero  point  is  not  absolute, 
and  is  only  stipulated  for  convenience.  Interval  data  values  can  be  added  and  subtracted, 
but  not  multiplied  or  divided.  Example  interval  data  sets  include  temperature  expressed 
on  the  Celsius  or  Fahrenheit  scale  (which  do  not  have  an  absolute  zero),  or  dates  on  a 
calendar. 
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Ratio  -  Ratio  data  incorporates  all  the  attributes  of  the  other  categories,  as  well  as 
an  “absolute”  zero  point,  which  indicates  a  complete  lack  of  the  value  being  measured. 
The  scale  is  consistent,  and  all  arithmetic  operations  can  be  performed,  including  the 
calculation  of  ratios  to  express  a  relationship  between  values.  Examples  include  age, 
temperature  expressed  on  the  Kelvin  scale  (which  does  have  an  absolute  zero),  and 
physical  measurements  (length,  weight,  height.) 

Summation  -  Summation  is  the  addition  of  a  set  of  numbers;  the  result  is  their 
sum.  Examples  of  “numbers”  to  be  summed  may  be  natural  numbers,  complex  numbers, 
matrices. 

Limits  on  the  range  of  measure:  Any  limits  on  input  or  output  of  the  measure. 

Rationale  for  the  measure:  Why  the  measure  was  selected  and  what  properties 
make  it  useful. 

Relevance  of  the  measure:  Circumstances  in  which  the  measure  would  contribute 
to  the  decision  process. 

Associated  measures:  Other  measures  which  may  be  used  in  conjunction  with  the 
measure,  or  which  must  be  used  with  it  to  appropriately  evaluate  the  issue. 
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(1.0)  Plan  C4I  SoS  Evaluation 

Planning  is  the  function  that  drives  the  content  of  an  evaluation.  The  planning 
function  identifies  the  resources  required  to  evaluate  the  performance  of  a  C4I 
SoS. 

(1.1)  Define  Problem  (Figure  48) 

Determination  of  the  SoS  capabilities  and  operating  conditions  to  be  considered  in 
the  SoS  evaluation. 


Figure  48.  Define  Problem. 

Functional  breakdown  of  Define  Problem  (1.1)  with  evaluation  measures 

(1.1.1)  Identify  SoS  Capabilities 

How  well  does  this  alternative  define  the  capabilities  of  the  SoS  to  be  evaluated? 
EM  #1:  Percentage  of  SoS  Capabilities  Identified 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  count  of  SoS  capabilities  identified 
by  the  alternative  by  total  SoS  capabilities.  Input  data  is  the  number  of 
SoS  capabilities. 

2.  Dimension  of  the  Measure: 
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Ratio  -  percentage  of  capabilities  identified. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent 

4.  Rationale  for  the  Measure: 

This  measure  directly  addresses  the  ability  of  the  alternative  to  effectively 
identify  capabilities. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternative  JC3M  solutions  under 
the  same  conditions. 

6.  Associated  Measures: 

Percentage  of  Operating  Conditions  Identified 


(1.1.1)  Identify  SoS  Operating  Conditions 

How  well  does  this  alternative  identify  the  conditions  under  which  the  SoS  will  be 
evaluated? 

EM  #1:  Percentage  of  Operating  Conditions  Identified 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  count  of  SoS  operating  conditions 
identified  by  the  alternative  JC3M  solution  by  total  SoS  operating 
conditions.  Input  data  is  the  list  of  SoS  operating  conditions,  including 
time  of  day,  location,  physical  environment,  and  distance. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  operating  conditions  identified. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. 

4.  Rationale  for  the  Measure: 
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This  measure  directly  addresses  the  ability  of  the  alternative  to  effectively 
identify  operating  conditions,  and  can  be  considered  an  indication  of  the 
flexibility  of  the  alternative  to  process  complex  SoS.  Understanding  the 
environment  in  which  the  SoS  will  be  used  will  support  testing  to  the  same 
requirements.  Does  this  alternative  identify  the  fleet  or  field  environment 
the  SoS  was  intended  to  perform  in? 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternative  JC3M  solutions  under 
the  same  conditions. 

6.  Associated  Measures: 

Percentage  of  SoS  Capabilities  Identified 
(1.2)  Define  Components  (Figure  49) 

Identify  the  System(s)  Under  Test  (SUT)  -  those  individual  systems  that  comprise 
the  SoS,  and  will  be  included  in  the 
evaluation. 


Figure  49.  Define  Components. 

Functional  breakdown  of  Define  Components  (1.2)  with  evaluation  measures 
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(1.2.1)  Identify  Systems  Under  Test 

How  well  does  this  alternative  identify  SUT  required  for  the  SoS  evaluation? 

EM  #1:  Percentage  of  SUT  Identified 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  count  of  SUT  identified  by  the 
alternative  by  total  SUT  count.  Input  data  is  the  number  of  SUT,  resident 
in  the  SoS,  required  for  the  evaluation. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  SUT  identified. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  identify  SUT. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

None. 

(1.2.2)  Identify  SUT  Capabilities 

How  well  does  this  alternative  identify  capabilities  the  SUT(s)  must  perform  to 
support  the  SoS  evaluation?  Function  1.1.1  identifies  the  SoS  Capabilities;  this  function 
identifies  the  capabilities  of  the  system(s)  participating  in  the  evaluation. 

EM  #1:  Percentage  of  SUT  Capabilities  Identified 

1 .  Definition  of  the  Measure: 
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This  EM  is  calculated  by  dividing  the  count  of  SUT  capabilities  identified 
by  the  alternative  by  total  SUT  capabilities.  Input  data  is  the  number  of 
SUT  capabilities. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  capabilities  identified. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  identify 
capabilities. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Percentage  of  SoS  capabilities  identified. 

(1.2.3)  Identify  SUT  Interfaces 

How  well  does  this  alternative  identify  interfaces  between  SUT  which  are 
required  to  support  the  SoS  evaluation? 

EM  #1:  Percentage  of  SUT  Interfaces  Identified 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  SUT  interfaces  identified  by  the 
alternative  by  the  total  SUT  interfaces.  Input  data  is  the  number  of  SUT 
interfaces. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  interfaces  identified. 

3.  Limits  on  the  range  of  Measure: 
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The  output  can  assume  any  value  from  0  to  100  percent. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  identify  interfaces. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions 

6.  Associated  Measures: 

None 

(1.3)  Define  Evaluation  Criteria  ( Figure  50) 

Determination  of  the  Resources  required  to  conduct  the  evaluation,  and  criteria 
used  when  evaluating  the  SoS. 


Figure  50.  Define  Evaluation  Criteria. 

Functional  breakdown  of  Define  Evaluation  Criteria  (1.3)  with  evaluation  measures 

(1.3.1)  Identify  Required  Resources 

How  well  does  this  alternative  identify  resources  (people,  time,  equipment,  etc.) 
required  to  conduct  the  SoS  evaluation? 

EM  #1:  Percentage  of  Required  Resources  Identified 
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1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  count  of  resources  identified  by  the 
alternative  JC3M  solution  by  total  resources  required.  Input  data  is  the 
number  of  resources  required  for  SoS  evaluation,  including  personnel, 
facilities,  C4I  systems,  training,  and  services. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  resources  identified. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  identify  required 
resources. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

None 

(1.3.2)  Define  Measures 

How  well  does  this  alternative  identify  measures  of  performance  when  evaluating 
the  SoS? 

EM  #1:  Percentage  of  Traceable  Measures 
1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  number  of  measures  (traceable  to 
stakeholder  requirements)  generated  by  the  alternative,  by  the  number  of 
measures  generated  by  the  alternative.  Input  data  is  the  list  of  stakeholder 
requirements. 
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2.  Dimension  of  the  Measure: 


3. 

4. 

5. 

6. 

EM# 

1. 


2. 

3. 

4. 

5. 


Ratio  -  percentage  of  traceable  measures  identified. 

Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. . 

Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  identify  measures. 
Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

Associated  Measures: 

None. 

2:  Number  of  Traceable  Measures 
Definition  of  the  Measure: 

Number  of  Traceable  Measures  is  calculated  by  summing  the  number  of 
measures,  traceable  to  stakeholder  requirements,  generated  by  the 
alternative.  Input  data  is  the  list  of  stakeholder  requirements. 

Dimension  of  the  Measure: 

Interval  -  total  of  traceable  measures  identified. 

Limits  on  the  range  of  Measure: 

Output  is  a  real  number  greater  than  or  equal  to  zero. 

Rationale  for  the  Measure: 

The  measure  addresses  the  ability  of  the  alternative  to  perform  a  core 
function  of  JC3M,  and  identify  measures. 

Relevance  of  the  Measure: 
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The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

None. 

(1.3.3)  Create  Test  Plan 

How  well  does  the  alternative  create  test  plans? 

EM  #1:  Quality  of  Test  Plan 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  assigning  an  overall  quality  level  to  the  test  plan 
produced  by  the  alternative.  Input  data  is  the  list  of  JC3M  internal  and 
external  documents. 

2.  Dimension  of  the  Measure: 

Ordinal  -  Low,  Medium,  High. 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  qualitative  ranking  of  the  outputs. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  create  effective  test 
plans. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions 

6.  Associated  Measures: 

None 

(1.4)  Ensure  Evaluation  Readiness  (Figure  51) 

Review  plans  and  assumptions  to  ensure  required  resources  are  available,  or  that  a 
remediation  plan  is  in  place.  Identification  of  shortfalls  will  determine  if  the  evaluation 
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can  continue,  or  if  additional  planning  is  required.  The  review  is  designed  to  ensure 
expectations  and  risks  are  correctly  identified.  Before  the  evaluation  begins,  all 
stakeholders  must  agree  that  requirements,  expectations,  and  risks  are  accurately 
identified,  conflicts  resolved  or  noted,  and  remediation  plans  are  in  place.  When  the 
review  is  completed,  measures  are  gathered  on  the  conduct  of  the  entire  planning  phase. 


Figure  5 1 .  Ensure  Evaluation  Readiness. 

Functional  breakdown  of  Ensure  Evaluation  Readiness  (1.4)  with  evaluation  measures 

(1.4.1)  Identify  Shortfalls 

How  well  does  this  alternative  identify  gaps  between  resources  required  to 
evaluate  the  SoS  and  the  resources  included  in  the  plan  for  SoS  evaluation? 

EM  #1:  Percentage  of  Shortfalls  Identified 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  number  of  resource  shortfalls 
identified  by  the  alternative,  by  the  total  number  of  shortfalls.  Input  data 
to  be  used  is  the  list  of  required  resources  and  known  shortfalls. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  resource  shortfalls  identified. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. 
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4.  Rationale  for  the  Measure: 


This  measure  addresses  the  ability  of  the  alternative  to  identify  missing  or 
incomplete  resources. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

None. 


(1.4.2)  Conduct  Review 

How  long  does  the  alternative  take  to  review  the  evaluation  plan  with  the 
stakeholders? 

EM  #1:  Review  Duration 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  summing  the  total  duration  of  the  evaluation  plan 
review. 

2.  Dimension  of  the  Measure: 

Ratio  -  time  in  hours. 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  real  number  greater  than  or  equal  to  zero. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  conduct  readiness 
reviews. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 
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6.  Associated  Measures: 


Percentage  of  Documents  Review  Quality  of  JC3M  Documents 
EM  #2:  Percentage  of  documents  reviewed 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  dividing  the  number  of  JC3M  internal  and 
external  documents  (to  included  SE  Artifacts)  reviewed  by  the  total 
number  of  JC3M  internal  and  external  documents. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  of  documents  reviewed. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  review  JC3M 
documents  effectively. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Review  Duration 

Quality  of  JC3M  Documents 
(1.4.3)  Planning  Results 

How  long  did  this  alternative  take  to  execute  the  planning  phase  take,  and  what 
were  the  results? 

EM  #  1 :  Hours  to  Plan  Evaluation 
1 .  Definition  of  the  Measure: 
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At  the  completion  of  the  review,  this  EM  is  calculated  by  summing  the 
labor  man-hours  required  to  plan  an  evaluation.  Input  data  is  the  duration 
of  each  Planning  task,  as  performed  by  the  alternative.  Note  that  this  data 
was  not  used  to  directly  compare  the  performance  of  alternatives.  Rather, 
this  data  was  used  in  the  LCCE  to  calculate  the  personnel  costs  associated 
with  performance  of  tasks  by  an  alternative. 

2.  Dimension  of  the  Measure: 

Summation  -  duration  of  evaluation  planning. 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  real  number  greater  than  or  equal  to  zero. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  labor  hours  used  in  the  planning  phase  of 
JC3M  system,  and  indirectly  addresses  the  duration  of  JC3M  system 
evaluations. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Days  to  Plan  Evaluation. 

Quality  of  Planning  Outputs. 

EM  #  2:  Days  to  Plan  Evaluation 

1 .  Definition  of  the  Measure: 

At  the  completion  of  the  review,  this  EM  is  calculated  by  summing  the 
days  required  to  plan  an  evaluation.  Input  data  is  the  duration  of  each 
Planning  task,  as  performed  by  the  alternative. 

2.  Dimension  of  the  Measure: 
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Ratio  -  duration  of  evaluation  planning  in  days. 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  real  number  greater  than  or  equal  to  zero. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  duration  of  the  planning  phase  of  JC3M 
system,  and  indirectly  addresses  the  duration  of  JC3M  system  evaluations. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Hours  to  Plan  Evaluation. 

Quality  of  Planning  Outputs. 

EM  #3:  Quality  of  Planning  Outputs 

1 .  Definition  of  the  Measure: 

At  the  completion  of  the  review,  this  EM  is  calculated  by  assigning  an 
overall  quality  level  to  the  planning  documents  produced  by  the 
alternative.  Input  data  is  the  list  of  JC3M  system  internal  and  external 
documents. 

2.  Dimension  of  the  Measure: 

Ratio  -  from  1  -  4  (Likert  Scale.) 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  qualitative  ranking  of  the  JC3M  internal  and  external 
documents. 

4.  Rationale  for  the  Measure: 
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This  measure  addresses  the  ability  of  the  alternative  to  create  effective 
JC3M  documents. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Hours  to  Plan  Evaluation 

Days  to  Plan  Evaluation 
(2.0)  Evaluate  Capability  of  C4I  SoS 

Test  agencies  will  use  JC3M  to  evaluate  the  performance  of  the  SoS.  JC3M  does 
not  include  new  processes,  and  instead  relies  on  existing  Joint  and  Service  processes  for 
the  conduct  of  C4I  SoS  evaluations.  Because  JC3M  relies  on  existing  processes,  this 
function  is  not  explicitly  modeled,  nor  are  there  defined  evaluation  measures. 

Modeling  and  simulation  is  highly  recommended  as  part  of  the  conduct  of  any 
C4I  SoS  evaluation. 
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(3.0)  Report  Results 

Test  and  acquisition  agencies  will  use  JC3M  to  report  the  results  of  an  evaluation 
because  the  existing  process  which  do  in  fact  clearly  indicate  pass  or  fail  based  on 
meeting  or  not  meeting  a  set  of  thresholds  can  no  longer  be  used.  Therefore,  the  reporting 
system  needs  to  be  different  to  reflect  the  results  in  a  form  of  “here  is  what  the  system 
does”  and  allow  the  end-user  to  say  if  it’s  good  enough. 

The  evaluation  and  the  planning  for  the  evaluation  do  not  focus  on  creating  the 
threshold  values  but  rather  focus  on  finding  and  defining  more  reasonable  evaluation 
measures  that  make  more  sense  to  an  end-user,  but  not  defining  “how  good  is  good.” 

(4.0)  Adaptability  (Figure  52) 


Figure  52.  Adaptability. 

Non-functional  breakdown  of  Adaptability  (4.0)  with  Evaluation  Measures 


(4.1)  Input  System  Flexibility 

How  well  does  the  alternative  support  varying  types  of  C4I  SoS? 

EM  #:  1  Types  of  C4I  SoS 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  recording  the  type  of  SoS  the  alternative  can 
evaluate.  Input  data  is  the  list  of  SoS  types. 

2.  Dimension  of  the  Measure: 

Nominal  -  Joint,  Service,  both. 

3.  Limits  on  the  range  of  Measure: 
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Output  is  a  list  of  SoS  types  supported  by  the  alternative. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  evaluate  varying 
types  of  C4I  SoS. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Number  ofC4I  SUT 
(4.2)  Input  System  Elasticity 

How  does  the  alternative  respond  when  the  SoS  evaluation  changes  scope? 

EM  #:  1  Elasticity  of  Labor 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  dividing  the  percent  change  in  systems  under  test  by 
the  percent  change  in  labor  hours  to  conduct  the  planning  phase  of  JC3M. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  change. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  plan  C4I  SoS  of 
varying  sizes. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 
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6.  Associated  Measures: 


Elasticity  of  Duration 
EM  #  2:  Elasticity  of  Duration 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  dividing  the  percent  change  in  systems  under  test  by 
the  percent  change  in  duration  to  conduct  the  planning  phase  of  JC3M. 

2.  Dimension  of  the  Measure: 

Ratio  -  percentage  change. 

3.  Limits  on  the  range  of  Measure: 

The  output  can  assume  any  value  from  0  to  100  percent 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  plan  C4I  SoS  of 
varying  sizes. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

Elasticity  of  Labor 
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(5.0)  Usability  (Figure  53) 


Figure  53.  Usability. 

Non-functional  breakdown  of  Usability  (5.0)  with  evaluation  measure. 

(5.1)  Implementation  of  JC3M 

What  resources  are  required  to  implement  the  alternative?  What  is  the  duration  of 
the  implementation  process,  i.e.  the  time  required  to  replace  the  baseline  process  with  the 
alternative? 

EM  #  1:  Time  to  implement  JC3M 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  summing  the  time  required  to  implement  the 
alternative.  Input  data  is  based  on  calculated  implementation  time. 

2.  Dimension  of  the  Measure: 

Ratio  -  duration  of  implementation. 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  real  number  greater  than  or  equal  to  zero. 

4.  Rationale  for  the  Measure: 
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This  measure  addresses  the  duration  of  implementation,  and  indirectly 
addresses  the  cost  of  implementing  the  alternative. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

None. 

(6.0)  Repeatability  (Figure  54) 

One  goal  of  JC3M  is  to  produce  the  same  outputs  for  each  SoS  evaluation.  If 
JC3M  consistently  produces  products  at  the  same  level  of  quality,  for  a  variety  of 
scenarios,  it  has  high  repeatability.  As  scenarios  vary,  what  is  the  effect  on  the  quality  of 
the  output  from  the  alternative? 

Repeatability 

6.0 


EM:  Variance  in  Quality 


Figure  54.  Repeatability. 

Non-functional  breakdown  of  Repeatability  (6.0)  with  evaluation  measure 
EM  #  1:  Variance  in  Quality 

1 .  Definition  of  the  Measure: 

This  EM  is  calculated  by  assigning  an  overall  quality  level  to  the  products 
produced  by  the  alternative  in  varying  scenarios.  This  EM  is  not  just 
measuring  the  quality  level,  but  the  difference  in  the  quality  level  between 
the  different  scenarios.  Input  data  is  the  list  of  JC3M  internal  and  external 
documents. 

2.  Dimension  of  the  Measure: 


186 


Ordinal  -  Low,  Medium,  High. 

3.  Limits  on  the  range  of  Measure: 

Output  is  a  qualitative  ranking  of  the  outputs. 

4.  Rationale  for  the  Measure: 

This  measure  addresses  the  ability  of  the  alternative  to  create  effective 
documents. 

5.  Relevance  of  the  Measure: 

The  measure  may  be  used  to  compare  alternatives  under  the  same 
conditions. 

6.  Associated  Measures: 

None. 
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APPENDIX  F.  CORE  IDEFO  DIAGRAMS 


FEDOS  IDEFO  Diagram 


SUT  Versions  Numbers 
Test  Dates 
5UT  List 

Stakeholder  Requirements 


List  of  Proposed  Tests 
Network  Arch  Diagrams 
List  of  Proposed  Eval  Metrics 


Test  Schedule 
System  Delivery  Dates  ! 

Staffing  Plan  1 


Elicit  Stakeholder 
Requirements  for 
C4ISP  Systems 
Configuration  B. . . 


Confirmed  Test  Dates 
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hfirmed  List 
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Generate  an 
Evaluation  Plan 
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.  r 

]  Team  Lead 
Log  Manager 


\ 

Config  Mgr/Qual  Cntrl 


F.3 


Develop  a  Test 
Plan  and  Test 
Procedures 


-►  Test  Plan 


-►  Test  Procedures 


\  Planning  Lead 

N 

Config  Mgr/Qual  Cntrl 
DCACT  Lead  "A" 

DCAT  Lead  "B" 

Tech  Lead 

test  Coord  Site  Lead 
Sr.  Tech  Writer 


Figure  55.  FEDOS  IDEFO  Diagram. 

This  diagram  illustrates  only  the  FEDOS  tasks  performed  in  planning  a  C4I  SoS  Evaluation.  Parentheses  (“tunnels”)  on  the  left  indicate 
stakeholder  inputs  or  previous  tasks.  Test  Plans  and  Procedures  connect  to  the  complete  FEDOS  C4I  SoS  evaluation  process. 
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MC3T  IDEFO  Diagram 


MC3M  Capabilities  Package  \ - ► 


Consolidated  Requirements  Review  (CRA) 


In  Band  and  Out-of-Band  Topologies 
Test  Cases/In  Band  and  Out-of-Band  Topologies 


Technical  Proposal 


Data  Analysis  Plan 
Data  Capture  Plan 

__Draft  In  Band  Topology 
Draft  Out  of  Band  Topology 
Test  Cases 

Topology  Review  Results 
MC3M  Technical  Proposal 


Event  Plan 


Data  Collection  and  Analysis  Lead  "A"  (DDACT) 

Logistics  Manager 


Configuration  Manager/Quality  Control 
Sr.  Technical  Writer 
technical  Lead 
test  Coordinator  Site  Lead 
Lead  C4I  Operator 
Planning  Lead 
Sr.  Program  Manager 


Figure  56.  MC3T  IDEFO  Diagram. 

This  diagram  illustrates  only  the  MC3T  tasks  performed  in  planning  a  C4I  SoS  Evaluation,  and  reflects  MC3T  naming  and  numbering 
conventions.  Tunnels  on  the  left  indicate  tasks  and  inputs  to  planning;  tunnels  on  the  right  are  used  to  indicate  personnel  inputs. 
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JTEM  CTM  IDEFO  Diagram 


Approved  AoA  Plan 


Capability  Dev,  Document  (CDD) 
Initial  Capabilities  Document  (ICD) 
Joint  Capabilities  Document  (JCD) 
Analytical  Baseline  (Scenario,  CONOPS,  Data) 
JOpsC  Family  (JOC,  JFC,  JIC) 
Other  Test  Plans  (D,  LF 
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Test  &  Evaluation  Strategy 
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Capability  Prod,  Document  (CPD) 
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Joint  DOTMLPF  Change  Recommendation  — 
Enterprise  JME  LVC  Foundation  Model  (JFM)  — 
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Test  Plan 


->  Joint  Capability  Evaluation  Report  (JCER) 
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Implement  LVC 
Distributed 

C7 

■“Test  Plan 

Environment 

Figure  57.  JTEM-CTM  IDEFO  Diagram. 

This  diagram  illustrates  only  the  JTEM  CTM  tasks  performed  in  planning  a  C4I  SoS  Evaluation,  utilizes  JTEM  CTM  naming  and  numbering 
conventions,  and  may  change  as  the  JTEM  CTM  matures. 


191 


FCB  IDEFO  Diagrams 


Figure  58.  FCB  IDEFO  Diagram. 

This  diagram  illustrates  only  the  FCB  tasks  performed  in  planning  a  C4I  SoS  Evaluation.  Tunnels  on  the  left  indicate  inputs  to  planning;  Final 
Test  Plan  on  the  right  was  proposed  for  use  with  existing  C4I  SoS  evaluation  processes.  “FCB  Measurements”  is  the  input  from  the  C2  FCB. 
This  is  in  comparison  to  the  “SCR  Measures”  input  seen  in  Figure  60. 
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FCB  Evaluation  Measurements 


Tech  Lead 


Figure  59.  FCB’s  Analyze  Measures  task 

This  figure  illustrates  only  FCB  task  1.3. 2.1  and  how  measures  proposed  by  the  FCB,  and  other  inputs,  would  be  analyzed  by  the  test 
organization,  and  converted  into  metrics  for  use  in  the  planning  of  a  C4I  SoS  evaluation.  1.3. 2.1  is  a  subset  of  FCB  task  1.3  “Define  Criteria” 
as  seen  in  Figure  59. 
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SCR  IDEFO  Diagrams 


Figure  60.  SCR  IDEFO  Diagram. 

This  diagram  shows  illustrates  only  the  SCR  tasks  performed  in  planning  a  C4I  SoS  Evaluation.  Tunnels  on  the  left  indicate  inputs  to 
planning;  Final  Test  Plan  on  the  right  was  proposed  for  use  with  existing  C4I  SoS  evaluation  processes.  “SCR  Measurements”  is  the  input  to 
this  task,  generated  within  the  test  organization.  This  is  in  comparison  to  the  “FCB  Measures”  input  seen  in  Figure  58. 
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Figure  61.  SCR’s  Define  Measures  task 

This  figure  is  a  subset  of  SCR  Task  1.3  in  Figure  61,  and  illustrates  a  detailed  view  ofhow  the  SCR  alternative  would  Define  Measures.  Inputs 
would  be  utilized  by  the  test  organization,  and  converted  into  metrics  for  use  in  the  planning  of  a  C4I  SoS  evaluation. 
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APPENDIX  G.  EVALUATION  MEASURE:  PERCENTAGE  OF 

TRACEABLE  MEASURES 


Percentage  of  Traceable  Measures  (PTM)  was  the  EM  for  the  Define  Measures 
(1.3.2)  function  of  the  JC3M  value  hierarchy.  The  objective  of  the  Define  Measures 
function  was  to  determine  how  well  an  alternative  identifies  measures  of  performance 
(MOP)  when  evaluating  the  SoS.  An  EM  should  not  be  confused  with  a  MOP.  EMs 
were  measures  created  by  this  team  to  gauge  functions  of  the  JC3M  system.  MOPs  are 
measures  used  to  gauge  performance  of  a  C4I  SoS. 

Definition  of  PTM: 

This  EM  was  calculated  by  dividing  the  number  of  measures  (traceable  to 
stakeholder  requirements)  generated  by  the  alternative,  by  the  number  of  measures 
generated  by  the  alternative. 


PTM  = 


#  Traceable  Measures  Created 
#  Total  Measures  Created 


However,  the  team  came  to  the  conclusion  that  it  was  not  feasible  to  calculate  the 
PTM  EM  as  it  was  defined  because  that  would  entail  exercising  each  of  the  alternative 
systems  and  developing  MOPs  for  each  alternative.  Therefore,  a  proxy  measure  was 
developed  that  could  serve  as  an  indicator  to  the  performance  of  Define  Measures 
function. 

Proxy  Definition  of  PTM: 

This  EM  was  calculated  by  taking  the  number  of  authoritative  sources  that  were 
considered  in  the  process  and  then  dividing  by  the  total  number  of  authoritative  sources 


available  for  the  SoS. 


PTM  = 


#  Authoritative  Sources  Considered 

#  Authoritatative  Sources  Available 


The  concept  was  that  analysts  performing  the  Define  Measures  function  should 
consider  all  available  and  appropriate  documentation  for  the  SoS.  Considering  all 
available  documentation  helped  ensure  that  all  requirements  and  testable  capabilities 
were  captured  in  the  process  and  subsequently  MOPs  were  defined  for  each.  The  team 
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assumed  that  considering  a  wider  set  of  authoritative  sources  would  yield  a  higher 
number  of  requirements  and  capabilities,  and  in  turn  provide  more  metrics  that  are 
traceable.  The  team  recognized  that  a  Program  Manager  (PM)  might  be  resistant  to  other 
sources  used  to  define  requirements  for  an  SoS  component  system,  when  a  CDD  or  ORD 
may  have  been  used  used  for  system-level  requirements. 

Figure  63  illustrates  how  measures  are  traceable  to  requirements  for  systems  in 
acquisition  programs.  However,  the  JC3M  problem  statement  is  to  consider  evaluating 
SoSs  that  have  been  constructed  or  assembled  with  existing  fielded  systems.  While 
component  systems  in  the  SoS  do  have  CDDs  or  ORDs,  the  assembled  SoS  does  not  have 
CDDs,  ORDs,  or  an  overarching  integrated  architecture.  This  fact  drives  the  need  to 
obtain  overall  SoS  requirements  and  measures  from  all  relevant  sources. 
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Figure  62.  Performance  Measure  Traceability. 

Performance  Measures  Trace  Back  to  System  Capabilities  and  Requirements  [From 
Meyer,  2007], 

Figure  63  illustrates  how  performance  measures  can  be  derived  using  an 
expanded  set  of  authoritative  sources.  The  next  section  describes  the  list  of  documents 
derived  using  various  DoD  Directives  and  Instructions. 
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Figure  63.  Fielded  IT  &  NSS  Interoperability  Process. 

Capability-focused,  effects-based  interoperability  process  for  addressing  operational 
warfighting  interoperability  and  supportability  issues  for  fielded  IT  and  NSS  [From  DoDI 
4630.8,2004:26], 
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Building  the  PTM  Denominator 

The  proxy  definition  for  PTM  states  the  total  number  of  authoritative  sources 
available  for  the  SoS  is  the  denominator  of  the  measure.  The  team  assembled  a  list  of 
relevant  documents  from  three  categories:  1)  reference  documents,  2)  system  level 
documents  and  3)  overarching  DoD  documents.  The  reference  documents  include  DoD 
Instructions  (DoDI),  DoD  Directives  (DoDD),  and  Combined  Joint  Chiefs  of  Staff 
Instructions  (CJCSI.)  The  reference  documents  apply  to  all  IT  and  NSS  and  provide 
specific  requirements  in  determining  interoperability  needs.  The  system  level  documents 
are  specific  to  any  given  system  under  test,  and  include  system  specific  capabilities 
documents.  Overarching  DoD  documents  include  Joint  documents  that  are  applicable 
across  Services  and  or  across  different  C4I  systems;  these  include  Joint  Concept 
documents,  Joint  Doctrine,  and  integrated  architectures, 
a.  Reference  Documents 


1 .  DoDD  5000. 1 ,  DoDI  5000.2 

2.  DoDD  4630.5  and  DoDI  4630.8 

3.  DoDI  5200.40 

4.  DoDD  8500. 01E 

5.  CJCSI  3170.01F 

6.  CJCSI  6212.01D 

7.  NCOWRM 


b.  Design  Artifacts 

1 .  JCIDS  Capabilities  Documents 

i)  Joint  Capabilities  Document  (JCD) 

ii)  Initial  Capabilities  Document  (ICD) 

iii)  Capabilities  Development  Document  (CDD)  and  Capabilities  Production 
Document  (CPD) 
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iv)  Capstone  Requirements  Document  (CRD) 

2.  System  Threat  Assessment  (STA) 

3.  Analysis  of  Alternatives  (AO  A) 

4.  Information  Support  Plan  (ISP) 

5.  Test  and  Evaluation  Master  Plan  (TEMP) 

6.  Test  Plans 

7.  Data  Management  and  Analysis  Plans  (DMAP) 

8.  Reports  from  T&E,  exercises,  and  readiness  reports 

9.  Combatant  Commanders  Input  and  Issues 

c.  Overarching  DoD  Documents 

1 .  Joint  Doctrine 

2.  Universal  Joint  Task  List  (UJTL),  Service  task  lists,  and  Mission  Essential  Task 
Lists  (METLs) 

3.  Joint  Operating  Concepts  (JOC) 

4.  Joint  Functional  Concepts  (JFC) 

Description  of  Applicability  of  Documents 

The  list  was  built  using  guidance  from  [DoDD  4630.5,  2004:  3]  which  states  that 
all  IT  and  NSS  interoperability  and  supportability  needs  shall  be  derived  using  Joint 
Operating  Concepts,  Joint  Functional  Concepts,  and  associated  integrated  architectures 
and  shall  be  updated  as  necessary  throughout  the  system's  life.  IT  and  NSS 
interoperability  and  supportability  needs,  for  a  given  capability,  shall  be  identified 
through  the  following: 

1.  The  Defense  Acquisition  System  (as  defined  in  the  5000  series  of  DoD  issuances) 

2.  The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  process. 
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3.  The  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  education, 

Personnel  and  Facilities  (DOTMLPF)  change  recommendation  process. 

In  addition,  Universal  Reference  Resources  (URRs)  are  resources  that  must  be 
consulted  when  building  integrated  architecture  products  for  IT  and  NSS.  URRs  are: 

.  .DoD  Architecture  Framework;  DoD  Core  Architecture  Data  Model;  Universal  Joint 
Task  List;  Technical  Reference  Model;  Global  Information  Grid  (GIG)  Architecture; 

DoD  Net-Centric  Data  Strategy;  DoD  Metadata  Registry;  NCOW  RM;  and  the  DISR” 
[DoDD  4630.5,  2004:  57], 

“Integrated  Architectures  shall  be  used  as  the  basis  for  assessment  and  analysis  to 
characterize  interoperability  needs  for  a  given  capability.  Solutions  to  the  identified  needs 
may  be  materiel  or  non-materiel  solution  sets,  or  both.  Interoperability  needs  shall  be 
documented  in  applicable  capabilities  documents  and  the  NR-KPP.  System  information 
and  interoperability  dependencies  and  supportability  requirements  shall  be  documented  in 
an  ISP.  Regardless  of  the  solution  (materiel  or  non-materiel),  interoperability  and 
supportability  shall  be  tested  and  verified  prior  to  operational  use  or  fielding” 

[DoDI  4630.8,  2004:  22], 

The  remainder  of  this  section  provides  more  detail  on  the  list  of  documents. 

DoDD  5000.1 

This  directive  applies  to  all  DoD  acquisition  programs.  DoD  policy  is  to  translate 
operational  needs  into  stable,  affordable  programs  (e.g.,  through  integrated  product  & 
process  development,  long-range  investment  plans,  risk  management,  use  of  a  “cost  as  an 
independent  variable”  approach,  performance  specifications),  acquire  quality  products 
(e.g.,  based  on  competition,  test  &  evaluation,  modeling  &  simulation,  past  performance, 
experience  in  domain,  mature  software  process),  organize  for  efficiency  &  effectiveness 
(e.g.,  streamlined  organization,  trained  acquisition  corps,  teams,  tailoring,  automated 
acquisition  information  corps,  teams,  tailoring,  automated  acquisition  information)  and 
make  use  of  test  and  evaluation  criteria  as  the  basis  of  managing  an  acquisition  program 
[DoDD  5000.1,2003:2], 
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Enclosure  1  to  the  directive  provides  additional  policies  on  independent 
operational  test  agencies,  information  assurance,  information  superiority,  integrated  test 
and  evaluation,  and  interoperability  [DoDD  5000.1,  2003:  7], 

DoDI  5000.2 

This  instruction  implements  DoDD  5000.1,  and  applies  primarily  to  defense 
technology  projects  and  acquisition  programs,  but  some  requirements  are  solely  for 
Major  Defense  Acquisition  Programs  and  Major  Automated  Information  Systems.  The 
instruction  “establishes  a  simplified  and  flexible  management  framework  for  translating 
mission  needs  and  technology  opportunities,  based  on  approved  mission  needs  and 
requirements,  into  stable,  affordable,  and  well-managed  acquisition  programs”  [DoDI 
5000.2,2003:  1], 

The  instruction  identifies  test  and  evaluation  procedures  for  major  acquisition 
programs  and  major  automated  information  systems  and  provides  procedures  for 
assessing  information  interoperability,  C4I  support,  and  information  assurance.  These 
procedure  are  limited  to  ACAT  I  or  IA  programs  and  special  interest  programs 
designated  by  the  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence  (ASD  C3I.)  The  instruction  also  establishes 
procedures  for  integrated  architectures  and  defines  the  responsibilities  of  a  PM  on 
development,  operational,  interoperability,  and  information  assurance  tests. 

DoDD  4630.5 

This  directive  defines  a  capability-focused,  effects-based  approach  for  IT  and 
NSS  interoperability  and  supportability  across  DoD.  The  directive  establishes  Net-Ready 
Key  Performance  Parameters  (NR-KPP)  to  assess  attributes  required  for  the  technical 
information  exchange  and  end-to-end  operational  effectiveness  of  the  exchange  [DoDD 
4630.5,2004:  1], 

DoDI  4630.8 

This  instruction  implements  a  capability- focused,  effects-based  approach  to  IT 
and  NSS  interoperability  and  supportability  throughout  DoD  in  order  to  ensure  their  life- 
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cycle  interoperability  and  supportability,  and  implements  NR-KPP  [DoDI  4630.8,  2004: 
25], 


DoDI  5200.40 

This  instruction  describes  policies,  processes,  and  the  importance  of  IT  security 
certification  and  accreditation  in  the  acquisition  of  IT  systems  [DoDI  5200.40,  1997:  1], 

DoDD  8500.1E 

This  directive  establishes  policy  for  Information  Assurance  (IA)  and  assigns 
responsibilities  to  achieve  DoD  IA  and  supports  the  evolution  to  network  centric  warfare 
[DoDD  8500. IE,  2002:  1], 

CJCSI  3170.01F 

This  instruction  describes  JCIDS  [CJCSI  3170. 01F,  2007]  and  the  identification, 
assessment,  and  prioritization  of  joint  military  capabilities  needs.  The  instruction 
implements  a  capabilities  based  approach  for  identifying  gaps  in  existing  capabilities  and 
developing  new  capabilities.  The  instruction  also  provides  guidelines  for  JCD,  ICD,  and 
CDD  production  and  use,  as  well  as  CPD  and  DOTMLPF  change  recommendations. 

CJCSI  6212.01D 

This  instruction  identifies  information  needs,  dependencies,  and  interfaces  for  IT 
and  NSS  programs  in  all  acquisitions,  focusing  on  net-readiness,  interoperability, 
information  supportability,  and  information  sufficiency  concerns.  The  instruction 
provides  guidance  on  the  use  of  the  Information  Support  Plan  (ISP)  and  interoperability 
evaluation  and  certification  procedures  [CJCSI  6212. 01D,  2006]. 

1.  Appendix  A  to  Enclosure  C  provides  guidance  on  the  use  of  the  ISP  (C4ISP) 
Assessment  tool  and  detailed  ISP  review  procedures  [CJCSI  6212. 01D,  2006:  C-A-l], 

2.  Appendix  B  to  Enclosure  C  Provides  guidelines  for  a  Tailored  Information 
Support  Plan  (TISP),  including  waivers,  content,  and  certification  [CJCSI  6212. 01D, 
2006:  C-B-l], 

3.  Enclosure  D  defines  the  submission  of  NR-KPP  documentation  and  describes 
JCIDS  and  ISP  Document  considerations;  JCD,  ICD,  CDD,  CPD,  and  ISP  production; 
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the  interoperability  certification  process;  Global  Information  Grid  (GIG)  Key  Interface 
Profile  (KIPs);  and  IA  requirements  [CJCSI  6212. 01D,  2006:  D-l], 

4.  Enclosure  E  provides  guidelines  for  Joint  interoperability  testing  and 
certification  [CJCSI  6212.01D,  2006:  E-l], 

Joint  Doctrine 

There  are  76  Joint  Doctrine  Publications  covering  Personnel,  Intelligence, 
Operations,  Logistics,  Plans,  and  Communications  Systems  support.  Applicable  Joint 
Doctrine  Publications  will  be  reviewed  depending  on  the  system  under  test. 

Universal  Joint  Task  List  (UJTL),  Service  task  lists,  and  Mission  Essential 
Task  Lists  (METLs) 

The  Universal  Joint  Task  List  (UJTL)  [CJCSM  3500.04D,  2005],  when 
augmented  with  Service  task  lists,  is  a  comprehensive  list  of  tasks,  conditions,  measures, 
and  criteria  supporting  all  levels  of  the  Department  of  Defense  in  executing  the  National 
Military  Strategy.  Mission  Essential  Task  Lists  (METL)  organize  tasks  by  mission. 

Thus  a  METL  is  a  compilation  of  tasks,  with  associated  standards  and  conditions.  The 
UJTL,  Service  Task  lists  METL  will  be  used  to  identify  tasks  that  the  system  under  test, 
as  part  of  the  SoS,  must  perform. 

Applicable  Joint  Operating  Concepts  (JOC)  and  Joint  Functional  Concepts 

(JFC) 

[DoDD  4630.5,  2004:  13]  states  all  IT  and  NSS  interoperability  and 
supportability  needs  shall  be  derived  using  Joint  Operating  Concepts  and  Joint  Functional 
Concepts.  JOCs  and  JFCs  applicable  to  the  C4I  SoS  under  evaluation  should  also  be 
reviewed.  There  are  currently  6  JOC  publications. 

1 .  Homeland  Defense  and  Civil  Support  [DoD  JOC  HDCS,  2006] 

2.  Deterrence  Operations  [DoD  JOC  DO,  2006] 

3.  Major  Combat  Operations  [DoD  JOC  MCO,  2006] 

4.  Stability  Operations  [DoD  JOC  SO,  2006] 

5 .  Irregular  Warfare  [DoD  JOC  IW,  2007] 

6.  Military  Support  to  Shaping  Operations  [DoD  JOC  MSSO,  2007] 

There  are  currently  8  JFC  publications: 
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1 .  Training  [DoD  JFC  T,  2007] 

2.  Force  Management  [DoD  JFC  FM,  2005] 

3.  Net  Centric  [DoD  JFC  NC,  2005] 

4.  Battlespace  Awareness  [DoD  JFC  BA,  2003] 

5.  Command  and  Control  [DoD  JFC  C2,  2007] 

6.  Force  Application  [DoD  JFC  FA,  2004] 

7.  Focused  Logistics  [DoD  JFC  FL,  2003] 

8.  Protection  [DoD  JFC  P,  2004] 

Determining  the  Score  for  the  PTM  Evaluation  Measure 

If  an  alternative  considers  every  document  in  the  comprehensive  list  of 
authoritative  documents  discussed  in  the  previous  section  it  receives  a  score  of  100%. 
Table  A-l  contains  the  comprehensive  list  of  authoritative  documents,  weights  for  each 
document,  and  the  score  for  each  alternative.  If  an  alternative  uses  a  document  in  its 
process  then  the  alternative  receives  the  complete  score  for  that  document. 

Based  on  a  number  of  25  authoritative  sources,  if  each  source  were  weighted 
equally  each  would  have  a  score  of  4.  However,  the  team  felt  that  there  were  some 
sources  that  would  contribute  more  to  the  Define  Measures  function  than  others. 
Therefore,  scores  were  derived  on  a  scale  of  2,  4,  and  8.  A  score  of  2  was  given  to 
sources  that  were  considered  by  the  team  to  have  the  least  contribution  to  deriving 
measurable  requirements  yet  still  essential.  Scores  of  4  were  given  to  sources  that  were 
cited  in  references  as  being  essential.  For  example,  DoDD  4630.5  [2004:  13]  states  that  it 
is  DoD  policy  that  interoperability  and  supportability  needs  shall  be  derived  using  Joint 
Operating  Concepts  and  Joint  Functional  Concepts.  Scores  of  8  were  given  to  all  the 
JCIDs  Capability  documents  which  were  deemed  as  the  most  important  sources  by  the 
team. 
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Authoritative  Source 

Score 

Baseline 

CTM 

MC3T 

FCB 

SCR 

a.  Reference  Documents  (Weight  incl.  1-7) 

16% 

n/a 

16% 

0% 

16% 

16% 

l.DoDD  5000.1  and  5000.2 

2% 

n/a 

2% 

0% 

2% 

2% 

2.  DoDD  4630.5  and  DoDI  4630.8 

2% 

n/a 

2% 

0% 

2% 

2% 

3.DoDI  5200.40 

2% 

n/a 

2% 

0% 

2% 

2% 

4.  DoDD  8500.1  (IA) 

2% 

n/a 

2% 

0% 

2% 

2% 

5.  CJCSI  3170  (J8) 

2% 

n/a 

2% 

0% 

2% 

2% 

6.  CJCSI  6212. 01D  (J6) 

2% 

n/a 

2% 

0% 

2% 

2% 

7.  NCOW  Reference  Model 

4% 

n/a 

4% 

0% 

4% 

4% 

b.  Design  Artifacts  (Weight  incl.  1-19) 

68% 

n/a 

60% 

56% 

56% 

60% 

1.  JCIDS  Documents  (Weight  incl.  i-iv) 

32% 

n/a 

32% 

32% 

32% 

32% 

i)  JCD 

8% 

n/a 

8% 

8% 

8% 

8% 

ii)  ICD 

8% 

n/a 

8% 

8% 

8% 

8% 

iii)  CDD  and  or  CPD 

8% 

n/a 

8% 

8% 

8% 

8% 

iv)  CRD 

8% 

n/a 

8% 

8% 

8% 

8% 

2.  STA 

4% 

n/a 

4% 

0% 

4% 

0% 

3.  AoA 

4% 

n/a 

0% 

0% 

4% 

0% 

4.  ISP 

4% 

n/a 

0% 

0% 

4% 

4% 

5.  TEMP 

8% 

n/a 

8% 

8% 

8% 

8% 

6.  Test  Plans 

4% 

n/a 

4% 

4% 

0% 

4% 

7.  Data  Management  and  Analysis  Plan 
(DMAP) 

4% 

n/a 

4% 

4% 

0% 

4% 

8.  Reports  from  T&E,  exercises,  and 
readiness  reports 

4% 

n/a 

4% 

4% 

0% 

4% 

9.  Combatant  Commanders  Input  and 
Issues 

4% 

n/a 

4% 

4% 

4% 

4% 

c.  Overarching  DoD  Docs.  (Weight  incl.  1- 
4) 

16% 

n/a 

16% 

16% 

16% 

16% 

1 .  Joint  Doctrine 

4% 

n/a 

4% 

4% 

4% 

4% 

2.  UJTL,  Service  task  lists,  METL 

4% 

n/a 

4% 

4% 

4% 

4% 

3.  JOC 

4% 

n/a 

4% 

4% 

4% 

4% 

4.  JFC 

4% 

n/a 

4% 

4% 

4% 

4% 

Total 

100% 

0% 

92% 

72% 

88% 

92% 

Table  26.  Percent  of  Traceable  Measures  Summary  Table 


PTM  Evaluation  Measure  detailed  results  by  alternative. 
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APPENDIX  H.  GLOSSARY  OF  TERMS 


Term 

Definition 

Action 

Something  done  (a  task  or  activity.) 

Activity 

A  function,  mission,  action,  or  collection  of  actions. 

Architecture 

A  framework  or  structure  that  portrays  relationships  among  all  the 
elements  of  the  subject  force,  system,  or  activity. 

Bandwidth 

The  difference  between  the  limiting  frequencies  of  a  continuous 
frequency  band  expressed  in  hertz  (cycles  per  second).  The  term 
bandwidth  is  also  loosely  used  to  refer  to  the  rate  at  which  data  can 
be  transmitted  over  a  given  communications  circuit.  In  the  latter 
usage,  bandwidth  is  usually  expressed  in  either  kilobits  per  second  or 
megabits  per  second. 

C4ISR 

Command,  control,  communications,  computers,  intelligence, 
surveillance,  and  reconnaissance 

Capability 

The  ability  to  execute  a  specified  course  of  action.  (A  capability  may 
or  may  not  be  accompanied  by  an  intention.) 

Certification 

Certification  is  the  process  of  confirming  that  a  system  or  component 
complies  with  its  specified  requirements  and  is  acceptable  for 
operational  use. 

Configuration 

Management 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  to:  (1)  identify  and  document  the  functional  and  physical 
characteristics  of  a  configuration  item;  (2)  control  changes  to  those 
characteristics;  and  (3)  record  and  report  changes  to  processing  and 
implementation  status. 

Elasticity 

A  measure  of  the  %  change  in  one  variable  (x  -  the  numerator)  with 
respect  to  another  variable  (y  -  the  denominator.) 

%  change  x  >  %  change  in  y  =  Elastic; 

%  change  in  x  <  %  change  in  y  =  Inelastic. 

Evaluation 

Measure 

Scale  to  measure  the  degree  that  we  attain  an  objective 

Goal 

Threshold  of  achievement 

Interoperability 

The  ability  of  systems,  units,  or  forces  to  provide  data,  information, 
materiel,  and  services  to  and  accept  the  same  from  other  systems, 
units,  or  forces,  and  to  use  the  data,  information,  materiel,  and 
services  so  exchanged  to  enable  them  to  operate  effectively  together. 

Measure  of 
Effectiveness 

A  criterion  used  to  assess  changes  in  system  behavior,  capability,  or 
operational  environment  that  is  tied  to  measuring  the  attainment  of  an 
end  state,  achievement  of  an  objective,  or  creation  of  an  effect. 
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Term 

Definition 

Measures  of 
Performance 

A  criterion  used  to  assess  friendly  actions  that  are  tied  to  measuring 
task  accomplishment. 

Network 

A  group  of  components  connected  by  communications. 

Objective 

Preferred  direction  of  attainment  of  an  evaluation  consideration 

Policy 

A  guiding  principle  or  constraint,  used  to  define  general  goals  and 
acceptable  criteria. 

Process 

Designated  series  of  actions  that  produces  some  outcome. 

Project 

Management 

Plan 

A  document  that  describes  the  background,  goals,  processes, 
resources,  and  schedule  used  to  perform  the  project's  tasks. 

Quality 

Assurance 

The  process  for  monitoring  a  project  to  ensure  that  defined  standards 
of  quality  are  met. 

Reliability 

The  probability  that  an  item  will  perform  its  function  under  stated 
conditions  of  use  and  maintenance  for  a  stated  measure  of  the  variant 
(time,  distance,  etc.) 

Requirement 

An  approved  need  to  achieve  a  capability,  which  is  used  in  turn  to 
accomplish  tasks. 

Stakeholder 

An  enterprise,  organization,  or  individual  having  an  interest  or  a  stake 
in  the  outcome  of  the  engineering  of  a  system. 

System 

A  combination  of  two  or  more  interrelated  pieces  of  equipment  (or 
sets)  arranged  in  a  functional  package  to  perform  an  operational 
function  or  to  satisfy  a  requirement. 

Systems 

Architecture 

The  organizational  structure  of  a  system  or  component,  their 
relationships,  and  the  principles  and  guidelines  governing  their  design 
and  evolution  over  time. 

Systems 

Engineering 

An  interdisciplinary  approach  and  means  to  enable  the  realization  of 
successful  systems.  It  focuses  on  defining  customer  needs  and 
required  functionality  early  in  the  development  cycle,  documenting 
requirements,  then  proceeding  with  design  synthesis  and  system 
validation  while  considering  the  complete  problem 

System 

Function 

An  activity  that  the  system  must  be  designed  to  perform  (Destroy 
targets)  and  an  evaluation  consideration  for  alternative  system 
designs 

210 


Term 

Definition 

Validation 

The  process  of  determining  the  degree  to  which  a  model  or 
simulation  is  an  accurate  representation  of  the  real  world  from  the 
perspective  of  the  intended  uses  of  the  model  or  simulation. 

Verification 

The  process  of  determining  that  a  model  or  simulation 
implementation  accurately  represents  the  developer’s  conceptual 
description  and  specifications. 
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